From 2216dc2d1dc44681750d44837bebea7ad961334c Mon Sep 17 00:00:00 2001 From: Michael Lieberman Date: Sat, 29 Aug 2026 15:01:44 -0400 Subject: [PATCH 1/2] docs: name skills + MCP as the canonical product path (closes #356) A new user opens `darnit --help` and sees `audit`, `run`, `serve`, `harness`, `install` at the same visual weight. Nothing tells them which is the front door. The intended product path -- skills invoking darnit's MCP tools inside a coding-agent client -- lives in maintainer heads and RFC-0001 today, not in a place a user would find it. Adds a "How to Use Darnit" section to README.md between Installation and Quick Start: 1. Product path: install the MCP config + skills into Claude Code / Claude Desktop / Cursor via `darnit install`, then invoke a skill (e.g. `/darnit-audit`) in the agent. The skill orchestrates the MCP tools; the user reasons conversationally. 2. CLI as dev/test scaffolding: a per-command table names what each of `serve`, `audit`, `run`, `harness` is for and what it is NOT for. Only `serve` is product-facing (the MCP server behind the skills). `audit`/`run`/`harness` are development, CI, and driver- testing tools respectively. 3. Direction of travel: cross-links RFC-0001, which formalizes the split (CLI becomes thin adapters around the harness runtime, not parallel entry points). Also re-frames the existing Quick Start intro: the Python code snippets shown are MCP tool signatures the `/darnit-audit` skill calls under the hood; in normal use the reader invokes the skill and never sees them directly. Preserves the snippets for the "embed darnit in your own tooling" and "debug the skill's orchestration" cases. `docs/getting-started/README.md` already has an "I want to use darnit with Claude Code" path -- unchanged; that section is consistent with the new README framing. --- README.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/README.md b/README.md index 3b9edb6a..f18b84b9 100644 --- a/README.md +++ b/README.md @@ -100,8 +100,48 @@ The tree-sitter discovery pipeline is tuned for **web-service shapes**. What wor If your project doesn't match a supported shape, the generator will still write a report — but it will likely show "Total findings: 0" because no entry points were discovered. That's a coverage gap on our side, not a clean bill of health. Expanding the query set is [tracked in our issue tracker](https://github.com/kusari-oss/darnit/issues?q=is%3Aissue+threat-model+coverage). +## How to Use Darnit + +Darnit's product path is **skills invoking MCP tools inside a coding agent** (Claude Code today; other clients coming). You do not call `darnit audit` on the CLI as a daily-use interface -- that command exists for local iteration on framework code and quick sanity checks, not as the front door. + +### The product path: skills + MCP + +1. Install darnit's MCP config and skills into your coding-agent client: + ```bash + darnit install # global (Claude Code default) + darnit install --project # per-project (.mcp.json + .claude/skills/) + darnit install --client claude-desktop + ``` +2. Restart the client. Skills appear as slash commands. +3. Invoke a skill in your agent -- for OpenSSF Baseline audits: + ``` + /darnit-audit + ``` + The skill orchestrates darnit's MCP tools behind the scenes: it runs the audit, handles PENDING_LLM consultations, calls remediation tools, generates attestations, and pulls in project context from `.project/project.yaml`. You reason about the results conversationally; you do not shell out. + +See [`docs/getting-started/using-skills.md`](docs/getting-started/using-skills.md) for the full skill catalog, install-target matrix, and multi-repo / profile-selection details. + +### The CLI: dev + debug scaffolding + +`darnit audit`, `darnit run`, and `darnit serve` all exist for a reason, but only `darnit serve` is a product-facing command (the MCP server the skills talk to). The other two are development tools: + +| Command | Intended use | Not for | +|---------|--------------|---------| +| `darnit serve` | The MCP server your coding agent connects to. Started automatically by the client's MCP integration. | Direct interactive use. | +| `darnit audit` | Single-run local check of one repo, without the coding-agent loop. Useful when iterating on framework or TOML controls. | Fleet auditing, remediation-in-conversation, PENDING_LLM handling. | +| `darnit run` | Batch execution for CI regression / parity testing. | Interactive daily-driver audits. | +| `darnit harness` | Full audit driver with in-band LLM dispatch and pluggable question resolvers -- the runtime the skills-via-MCP path uses under the hood. Reachable from the CLI for testing the driver itself. | Daily-driver interactive use (invoke via a skill instead). | + +If you're evaluating darnit and just want to see something happen: `uv run darnit audit /path/to/repo` will print a report. If you're actually using darnit on real projects: install the skills. + +### Direction of travel + +RFC-0001 (`docs/rfcs/0001-core-rearchitecture.md`) formalizes this split: the harness driver (Stage 1+) is the runtime that skills call into, and CLI commands become thin adapters around the harness rather than parallel entry points. When Stage 1 fully lands, the skills path gets more capable (in-band LLM dispatch, question resolvers) and the CLI stays at parity for the sanity-check use case. + ## Quick Start +The examples below show the MCP tool signatures that the `/darnit-audit` skill calls under the hood. In normal use you invoke the skill and never see these directly. They're useful for embedding darnit in your own tooling or debugging what the skill orchestrates. + ### Run an Audit ```python From 032ac51ffa718b27e6ec4a99faa8afdb9f03ebb6 Mon Sep 17 00:00:00 2001 From: Michael Lieberman Date: Tue, 1 Sep 2026 21:01:41 -0400 Subject: [PATCH 2/2] feat(stores): local-fs and user-local outside-repo backends (feature 034) Complements feature 033 (#396) with two new filesystem-backed `Store` backends inside darnit-core that write outside the audited repository: * `local-fs`: takes a config-driven `root` (absolute, ~-expanded, or $VAR-templated). $VAR interpolation uses `missing="raise"` mode so a typo is a hard KeyError at store construction, not a silent empty expansion. * `user-local`: resolves `root` from platform conventions (XDG on Linux, ~/Library/{Application Support,Caches}/darnit on macOS, %LOCALAPPDATA%\darnit\{Data,Cache}\ on Windows). No `root` config needed; explicit `root` kwargs are warn-and-ignored. Both selectable via `.baseline.toml`: [stores.attestation] backend = "local-fs" root = "$DARNIT_ATT_ROOT" [stores.report] backend = "user-local" The OSPO-leader consolidation use case (US1): env-var interpolation lets 30 repos share `root = "$DARNIT_ATT_ROOT"` in their .baseline.toml files with one per-machine env var doing the routing. No new config layer needed. Every successful outside-repo write emits one info-level log line to `darnit.stores.local` naming the backend, artifact kind, and resolved absolute path (FR-015 / SC-009). The pre-feature in-repo defaults emit zero lines to that logger, so zero-config audits stay log-silent. Design highlights: * No new runtime dependencies (FR-014). platform_paths.py hand-rolls XDG / macOS / Windows conventions in ~90 lines rather than pulling in `platformdirs`. * No `Store` Protocol methods added -- both backends satisfy the existing feature-033 Protocols by delegating to the in-repo classes after root resolution. * `.project/project.yaml` stays in the repo (FR-009). Neither backend is registered under `darnit.stores.project`; `[stores.project] backend = "user-local"` raises `StoreNotInstalled` at `resolve_stores` time before any control runs. * Path traversal is impossible by construction (SC-005). The shared `_sanitize_filename` regex from feature 033 handles it; a bundle_id like `"../../etc/foo"` produces a filename inside `root`, not a path escape. * Cross-filesystem cache writes are safe (R-004). The delegate already writes the tempfile into the target's directory, so os.replace never crosses filesystems. * Feature 033's US2 zero-config test passes unchanged (SC-003). New test_us4_zero_config_local.py extends the invariant to the new backends. * Two-step SC-008 error surface: LocalFs*/UserLocal* wrap the delegate's StoreOperationError with a message that names the backend, kind, and resolved path so operators can correlate the failure to their config. Spec artifacts under specs/034-local-output-store/: * spec.md: 4 user stories, 15 FRs, 9 SCs, 3-question clarify session * plan.md: constitution check PASS, project structure decision * research.md: 5 resolved research items (R-001 through R-005) * data-model.md: 5 entities including the info-log format * contracts/local-fs.md and contracts/user-local.md * quickstart.md: 3 worked examples * tasks.md: 41 tasks in 7 phases, all completed except T015 (deferred to a future PR because reimplementing .project/-prefix-free YAML I/O was scope creep) and T041 (Windows CI stretch goal) Test totals: baseline 97 stores tests -> 135 (+38 new). Full workspace sweep: 3061 pass, 26 skip, 0 fail. Ruff clean repo-wide. validate_sync clean. Structure and no-new-deps guards pass. --- .specify/feature.json | 2 +- CLAUDE.md | 2 +- docs/plugin-authoring/stores.md | 143 +++++++++++ packages/darnit/pyproject.toml | 22 ++ .../src/darnit/stores/defaults/__init__.py | 16 ++ .../src/darnit/stores/defaults/local_fs.py | 203 +++++++++++++++ .../darnit/stores/defaults/platform_paths.py | 93 +++++++ .../src/darnit/stores/defaults/user_local.py | 124 ++++++++++ .../checklists/requirements.md | 37 +++ .../contracts/local-fs.md | 102 ++++++++ .../contracts/user-local.md | 90 +++++++ specs/034-local-output-store/data-model.md | 161 ++++++++++++ specs/034-local-output-store/plan.md | 157 ++++++++++++ specs/034-local-output-store/quickstart.md | 146 +++++++++++ specs/034-local-output-store/research.md | 126 ++++++++++ specs/034-local-output-store/spec.md | 207 ++++++++++++++++ specs/034-local-output-store/tasks.md | 184 ++++++++++++++ tests/darnit/stores/test_local_fs_backend.py | 233 ++++++++++++++++++ tests/darnit/stores/test_local_fs_helpers.py | 96 ++++++++ .../darnit/stores/test_local_fs_isolation.py | 80 ++++++ tests/darnit/stores/test_local_fs_logging.py | 126 ++++++++++ tests/darnit/stores/test_platform_paths.py | 132 ++++++++++ .../stores/test_us4_zero_config_local.py | 54 ++++ .../darnit/stores/test_user_local_backend.py | 149 +++++++++++ 24 files changed, 2683 insertions(+), 2 deletions(-) create mode 100644 packages/darnit/src/darnit/stores/defaults/local_fs.py create mode 100644 packages/darnit/src/darnit/stores/defaults/platform_paths.py create mode 100644 packages/darnit/src/darnit/stores/defaults/user_local.py create mode 100644 specs/034-local-output-store/checklists/requirements.md create mode 100644 specs/034-local-output-store/contracts/local-fs.md create mode 100644 specs/034-local-output-store/contracts/user-local.md create mode 100644 specs/034-local-output-store/data-model.md create mode 100644 specs/034-local-output-store/plan.md create mode 100644 specs/034-local-output-store/quickstart.md create mode 100644 specs/034-local-output-store/research.md create mode 100644 specs/034-local-output-store/spec.md create mode 100644 specs/034-local-output-store/tasks.md create mode 100644 tests/darnit/stores/test_local_fs_backend.py create mode 100644 tests/darnit/stores/test_local_fs_helpers.py create mode 100644 tests/darnit/stores/test_local_fs_isolation.py create mode 100644 tests/darnit/stores/test_local_fs_logging.py create mode 100644 tests/darnit/stores/test_platform_paths.py create mode 100644 tests/darnit/stores/test_us4_zero_config_local.py create mode 100644 tests/darnit/stores/test_user_local_backend.py diff --git a/.specify/feature.json b/.specify/feature.json index 868fb396..c600a102 100644 --- a/.specify/feature.json +++ b/.specify/feature.json @@ -1 +1 @@ -{"feature_directory": "specs/033-pluggable-stores"} +{"feature_directory": "specs/034-local-output-store"} diff --git a/CLAUDE.md b/CLAUDE.md index fe031eae..50583300 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -381,5 +381,5 @@ else: For additional context about technologies to be used, project structure, shell commands, and other important information, read the current plan: -[`specs/033-pluggable-stores/plan.md`](specs/033-pluggable-stores/plan.md) +[`specs/034-local-output-store/plan.md`](specs/034-local-output-store/plan.md) diff --git a/docs/plugin-authoring/stores.md b/docs/plugin-authoring/stores.md index 12e6c32f..5f65cdd1 100644 --- a/docs/plugin-authoring/stores.md +++ b/docs/plugin-authoring/stores.md @@ -170,3 +170,146 @@ audit in a process that actually uses the artifact class you back -- never to zero-config runs, never to audits that skip your kind. Keep `__init__` cheap and open connections on first `write` if the backend is expensive to establish. + +--- + +# Writing artifacts outside the repo (feature 034) + +Feature 034 ships two additional filesystem-backed backends inside +darnit-core alongside the in-repo defaults. Both are selectable from +`.baseline.toml` under any `[stores.]` block; both write outside +the audited repository. They exist because the in-repo defaults land +attestations, reports, and audit-cache under `/.darnit/`, which +is often not where an operator wants them (backups, CI artifact +directories, XDG-idiomatic locations). + +## `local-fs`: arbitrary root path + +Points a store at any local filesystem path. Config: + +```toml +[stores.attestation] +backend = "local-fs" +root = "/absolute/path" # or "~/subpath", or "$VAR/subpath" +``` + +Path resolution runs in this order at store construction: + +1. `$VAR` interpolation via darnit's env-subst helper. A missing + variable raises `KeyError` immediately -- a typo does NOT silently + expand to `""`. This is deliberate: `root` is a compliance-critical + config value; loud failure beats surprise empty writes. +2. `~` expansion via `os.path.expanduser`. +3. Absolute `Path.resolve()`. + +Directory creation is deferred to the first write. Filename +sanitization is inherited from the in-repo default -- a bundle_id +containing `../../etc/foo` produces a sanitized filename inside +`root`, not a path escape. + +### Example: CI runner with persistent cache + report artifacts + +```toml +[stores.cache] +backend = "local-fs" +root = "$RUNNER_CACHE_DIR/darnit" + +[stores.report] +backend = "local-fs" +root = "$RUNNER_ARTIFACTS_DIR/darnit-reports" +``` + +- First-run: cache write goes to `$RUNNER_CACHE_DIR/darnit/`; next-run + cache read hits because the runner restored the cache directory + between jobs. +- Reports land where the runner's artifact-upload step already looks. + Markdown / JSON / SARIF each become their own file under that root; + one info log line per format. + +### Multi-repo templating + +Darnit has no org-level or user-level config file. If you want the +same `[stores.]` block active on 30 repos, use one of: + +1. **Env-var interpolation (easiest)**. Keep `root = "$DARNIT_ATT_ROOT"` + in every repo's `.baseline.toml`. Set `DARNIT_ATT_ROOT` once per + machine (shell profile, systemd unit, CI runner env). The 30 repos + share the destination without duplicating the literal path. +2. **CI/CD templating**. Your workflow rewrites `.baseline.toml` before + invoking `darnit audit`. +3. **Cookiecutter / repo-init tool**. One-shot copy the block into + each repo when you first onboard it. + +### The `.project/` layer (FR-009) + +Neither `local-fs` nor `user-local` is registered under +`darnit.stores.project`. `.project/project.yaml` is the CNCF +`.project/` spec's canonical repo-committable artifact and stays in +the repo by design. If you write `[stores.project] backend = "local-fs"` +in `.baseline.toml`, `resolve_stores` raises `StoreNotInstalled` +before any control runs -- the misconfiguration surfaces at audit +start, not in a confusing runtime failure. Redirecting project state +outside the repo means governance tooling can no longer find it, so +this is not supported. + +## `user-local`: platform-conventional root + +Points a store at the platform-idiomatic user-scoped location. No +`root` config needed. Example: + +```toml +[stores.attestation] +backend = "user-local" + +[stores.report] +backend = "user-local" + +[stores.cache] +backend = "user-local" +``` + +Resolved paths per platform: + +| Platform | Attestations / Reports (data) | Cache | +|---|---|---| +| Linux (XDG defaults) | `~/.local/share/darnit/...` | `~/.cache/darnit/...` | +| Linux (`XDG_DATA_HOME=/X`) | `/X/darnit/...` | (see `XDG_CACHE_HOME`) | +| macOS | `~/Library/Application Support/darnit/...` | `~/Library/Caches/darnit/...` | +| Windows | `%LOCALAPPDATA%\darnit\Data\...` | `%LOCALAPPDATA%\darnit\Cache\...` | +| Unknown | XDG fallback (same as Linux) | XDG fallback | + +Passing a `root` kwarg to a `user-local` backend logs a warning and +uses the platform default anyway. Less disruptive than a hard error +for operators who copied a snippet from a `local-fs` example. + +## Logging + +Every successful outside-repo write emits one info-level log line to +the `darnit.stores.local` logger: + +``` +INFO darnit.stores.local: wrote attestation (local-fs): /home/mike/darnit-attestations/acme-widget-baseline-attestation.intoto.json +``` + +The in-repo `Filesystem*Store` defaults do NOT emit to this logger, so +zero-config audits stay log-silent under `darnit.stores.local`. + +## Zero-config unchanged + +If you do NOT add `[stores.*]` blocks, artifacts continue to land in +`/.darnit/` exactly as before this feature. `local-fs` and +`user-local` are opt-in. + +## Troubleshooting + +- **`KeyError: DARNIT_ATT_ROOT`**: the env var referenced in `root` + isn't set. `local-fs` fails fast on missing vars by design. Export + the variable or use a literal path. +- **`StoreOperationError: [local-fs attestation @ ...]`**: the + resolved `root` is unwritable, the disk is full, or the file is + locked. The error message names the backend, kind, and resolved path + so the operator can correlate. Darnit does NOT silently fall back + to the in-repo default. +- **File landed with `_`s in the name**: your identifier contained + filesystem-unsafe characters. The store sanitized them to prevent + path traversal. Rename the caller's identifier for cleaner output. diff --git a/packages/darnit/pyproject.toml b/packages/darnit/pyproject.toml index 7b11badb..1acc5d34 100644 --- a/packages/darnit/pyproject.toml +++ b/packages/darnit/pyproject.toml @@ -60,6 +60,28 @@ darnit = "darnit.cli:main" # points are logged and skipped at discovery time, never crash the harness. interactive_terminal = "darnit.harness.interactive_resolver:build" +# Feature 034: outside-repo filesystem store backends. Both `local-fs` (arbitrary +# root path via TOML config) and `user-local` (platform-conventional root, XDG / +# Apple / LOCALAPPDATA) ship in darnit-core alongside the in-repo defaults. +# `user-local` is deliberately NOT registered under `darnit.stores.project` per +# FR-009: `.project/project.yaml` must stay in the repo. +[project.entry-points."darnit.stores.attestation"] +local-fs = "darnit.stores.defaults.local_fs:LocalFsAttestationStore" +user-local = "darnit.stores.defaults.user_local:UserLocalAttestationStore" + +[project.entry-points."darnit.stores.report"] +local-fs = "darnit.stores.defaults.local_fs:LocalFsReportStore" +user-local = "darnit.stores.defaults.user_local:UserLocalReportStore" + +[project.entry-points."darnit.stores.cache"] +local-fs = "darnit.stores.defaults.local_fs:LocalFsAuditCacheStore" +user-local = "darnit.stores.defaults.user_local:UserLocalAuditCacheStore" + +# NOTE: `user-local` is deliberately NOT registered under +# `darnit.stores.project` per FR-009 -- `.project/project.yaml` stays +# in the repo. `[stores.project] backend = "user-local"` MUST raise +# `StoreNotInstalled` at `resolve_stores` time. + [project.optional-dependencies] attestation = [ "sigstore>=3.0.0", diff --git a/packages/darnit/src/darnit/stores/defaults/__init__.py b/packages/darnit/src/darnit/stores/defaults/__init__.py index aab2b5a7..a320745b 100644 --- a/packages/darnit/src/darnit/stores/defaults/__init__.py +++ b/packages/darnit/src/darnit/stores/defaults/__init__.py @@ -9,12 +9,28 @@ from darnit.stores.defaults.attestation import FilesystemAttestationStore from darnit.stores.defaults.cache import FilesystemAuditCacheStore +from darnit.stores.defaults.local_fs import ( + LocalFsAttestationStore, + LocalFsAuditCacheStore, + LocalFsReportStore, +) from darnit.stores.defaults.project import FilesystemProjectStateStore from darnit.stores.defaults.report import FilesystemReportStore +from darnit.stores.defaults.user_local import ( + UserLocalAttestationStore, + UserLocalAuditCacheStore, + UserLocalReportStore, +) __all__ = [ "FilesystemAttestationStore", "FilesystemAuditCacheStore", "FilesystemProjectStateStore", "FilesystemReportStore", + "LocalFsAttestationStore", + "LocalFsAuditCacheStore", + "LocalFsReportStore", + "UserLocalAttestationStore", + "UserLocalAuditCacheStore", + "UserLocalReportStore", ] diff --git a/packages/darnit/src/darnit/stores/defaults/local_fs.py b/packages/darnit/src/darnit/stores/defaults/local_fs.py new file mode 100644 index 00000000..f9328207 --- /dev/null +++ b/packages/darnit/src/darnit/stores/defaults/local_fs.py @@ -0,0 +1,203 @@ +"""Outside-repo filesystem-backed store variants (feature 034). + +Complements the in-repo :mod:`~darnit.stores.defaults.attestation`, +:mod:`~darnit.stores.defaults.report`, :mod:`~darnit.stores.defaults.cache`, +and :mod:`~darnit.stores.defaults.project` defaults from feature 033. +Each ``LocalFs*Store`` here takes a config-driven ``root`` (absolute, +``~``-relative, or ``$VAR``-templated) and delegates I/O to the matching +in-repo class. + +Concrete classes are defined in later phases; this module ships the +shared helpers first (T003): + +* :func:`_resolve_root_config` -- runs the R-003 chain + (``$VAR`` interpolation with ``missing="raise"``, ``~`` expansion, + absolute ``resolve()``). +* :func:`_log_wrote` -- emits the one-line INFO message required by + FR-015 for every successful outside-repo write. +""" + +from __future__ import annotations + +import os +from pathlib import Path +from typing import Any + +from darnit.core.env_subst import substitute_dollar_vars +from darnit.core.logging import get_logger +from darnit.stores.errors import StoreOperationError + +from .attestation import ( + _CONTENT_TYPE_EXT, + FilesystemAttestationStore, + _sanitize_filename, +) +from .cache import FilesystemAuditCacheStore +from .report import FilesystemReportStore + +# `get_logger("stores.local")` prefixes with "darnit." internally, +# yielding the fully-qualified logger name `darnit.stores.local`. +_LOCAL_LOGGER_NAME = "darnit.stores.local" +logger = get_logger("stores.local") + + +def _resolve_root_config(root: str | Path) -> Path: + """Turn a raw ``root`` config value into an absolute :class:`~pathlib.Path`. + + Applied in order (data-model E-001, research R-003): + + 1. If ``root`` is already a :class:`~pathlib.Path`, return it as-is + (test-only shortcut). + 2. ``substitute_dollar_vars(root, missing="raise")`` -- a typo in + a ``$VAR`` reference is a hard :class:`KeyError` at construction, + not a silent empty expansion (research R-003). + 3. :func:`os.path.expanduser` -- expand ``~`` and ``~user``. + 4. :meth:`~pathlib.Path.resolve` -- to canonical absolute form for + I/O and for logging. + + Directory creation is deferred to first write (matches the + in-repo defaults from feature 033). + """ + if isinstance(root, Path): + return root + if not isinstance(root, str): + raise TypeError( + f"root must be str or pathlib.Path, got {type(root).__name__}" + ) + substituted = substitute_dollar_vars(root, missing="raise") + expanded = os.path.expanduser(substituted) + return Path(expanded).resolve() + + +def _log_wrote(kind_tag: str, backend: str, resolved_path: Path) -> None: + """Emit the FR-015 info log line for a successful outside-repo write. + + Args: + kind_tag: ``"attestation"``, ``"report:markdown"`` | + ``"report:json"`` | ``"report:sarif"``, or ``"cache"``. + backend: ``"local-fs"`` or ``"user-local"``. + resolved_path: The absolute on-disk path the artifact was written to. + """ + logger.info("wrote %s (%s): %s", kind_tag, backend, str(resolved_path)) + + +class LocalFsAttestationStore: + """``AttestationStore`` variant writing to a config-driven ``root``. + + Delegates I/O to :class:`FilesystemAttestationStore` after resolving + ``root`` via :func:`_resolve_root_config`. Wraps the delegate's + ``StoreOperationError`` with additional context (backend name, + artifact kind, resolved target path) so SC-008's error surface + contract is satisfied. Emits the FR-015 info log line after every + successful write. + """ + + _BACKEND_NAME = "local-fs" + _KIND_TAG = "attestation" + + def __init__(self, root: str | Path, **_: object) -> None: + self._root = _resolve_root_config(root) + self._delegate = FilesystemAttestationStore(self._root) + + def _target_for(self, bundle_id: str, content_type: str) -> Path: + ext = _CONTENT_TYPE_EXT.get(content_type, ".bin") + return self._root / f"{_sanitize_filename(bundle_id)}{ext}" + + def write( + self, bundle_id: str, bundle_bytes: bytes, content_type: str + ) -> None: + target = self._target_for(bundle_id, content_type) + try: + self._delegate.write(bundle_id, bundle_bytes, content_type) + except StoreOperationError as err: + raise StoreOperationError( + f"[{self._BACKEND_NAME} {self._KIND_TAG} @ {target}] {err}" + ) from err + _log_wrote(self._KIND_TAG, self._BACKEND_NAME, target) + + def close(self) -> None: + self._delegate.close() + + +class LocalFsReportStore: + """``ReportStore`` variant writing to a config-driven ``root``. + + Delegates to :class:`FilesystemReportStore` after root resolution. + Emits one FR-015 info log per format written (`report:markdown`, + `report:json`, `report:sarif`) so multi-format audit runs are + self-documenting. + """ + + _BACKEND_NAME = "local-fs" + + def __init__(self, root: str | Path, **_: object) -> None: + self._root = _resolve_root_config(root) + self._delegate = FilesystemReportStore(self._root) + + def _target_for(self, report_id: str, ext: str) -> Path: + return self._root / f"{_sanitize_filename(report_id)}{ext}" + + def _write(self, report_id: str, ext: str, content: str, tag: str) -> None: + target = self._target_for(report_id, ext) + try: + self._delegate._write(report_id, ext, content) + except StoreOperationError as err: + raise StoreOperationError( + f"[{self._BACKEND_NAME} {tag} @ {target}] {err}" + ) from err + _log_wrote(tag, self._BACKEND_NAME, target) + + def write_markdown(self, report_id: str, content: str) -> None: + self._write(report_id, ".md", content, "report:markdown") + + def write_json(self, report_id: str, content: str) -> None: + self._write(report_id, ".json", content, "report:json") + + def write_sarif(self, report_id: str, content: str) -> None: + self._write(report_id, ".sarif", content, "report:sarif") + + def close(self) -> None: + self._delegate.close() + + +class LocalFsAuditCacheStore: + """``AuditCacheStore`` variant writing to a config-driven ``root``. + + Delegates to :class:`FilesystemAuditCacheStore` after root resolution. + Best-effort per the Protocol contract: `write` MUST NOT raise on + backend failure. The info log line is only emitted when the write + actually succeeded (i.e., the target file exists post-write); a + swallowed failure produces no info-level log line, matching FR-015's + "successful write" clause. + """ + + _BACKEND_NAME = "local-fs" + _KIND_TAG = "cache" + + def __init__(self, root: str | Path, **_: object) -> None: + self._root = _resolve_root_config(root) + self._delegate = FilesystemAuditCacheStore(self._root) + + def _target_for(self, cache_key: str) -> Path: + return self._root / f"{_sanitize_filename(cache_key)}.json" + + def read(self, cache_key: str) -> dict[str, Any] | None: + return self._delegate.read(cache_key) + + def write(self, cache_key: str, envelope: dict[str, Any]) -> None: + target = self._target_for(cache_key) + self._delegate.write(cache_key, envelope) + # FR-015: log only on successful write. Delegate swallows OSError + # to a warning; check post-write whether the file actually landed. + if target.exists(): + _log_wrote(self._KIND_TAG, self._BACKEND_NAME, target) + + def close(self) -> None: + self._delegate.close() + + +__all__ = [ + "LocalFsAttestationStore", + "LocalFsAuditCacheStore", + "LocalFsReportStore", +] diff --git a/packages/darnit/src/darnit/stores/defaults/platform_paths.py b/packages/darnit/src/darnit/stores/defaults/platform_paths.py new file mode 100644 index 00000000..1b12ff24 --- /dev/null +++ b/packages/darnit/src/darnit/stores/defaults/platform_paths.py @@ -0,0 +1,93 @@ +"""Platform-conventional path resolution for the ``user-local`` backend. + +Feature 034 T023. Concrete implementations of the four helpers whose +skeleton was scaffolded in T004. + +Platform dispatch (research R-001): + +* Linux (and unknown platforms): XDG Base Directory spec. + Data: ``${XDG_DATA_HOME:-$HOME/.local/share}/darnit`` + Cache: ``${XDG_CACHE_HOME:-$HOME/.cache}/darnit`` +* macOS (``platform.system() == "Darwin"``): + Data: ``~/Library/Application Support/darnit`` + Cache: ``~/Library/Caches/darnit`` +* Windows (``platform.system() == "Windows"``): + Data: ``%LOCALAPPDATA%\\darnit\\Data`` + Cache: ``%LOCALAPPDATA%\\darnit\\Cache`` + +Unknown platforms fall through to the XDG branch as the safest +heuristic. FR-014 forbids new runtime dependencies, so `platformdirs` +is deliberately not used. +""" + +from __future__ import annotations + +import os +import platform +from pathlib import Path + +from darnit.core.logging import get_logger + +logger = get_logger("stores.platform_paths") + +__all__ = [ + "xdg_data_home", + "xdg_cache_home", + "user_data_root", + "user_cache_root", +] + + +def xdg_data_home() -> Path: + """Return ``$XDG_DATA_HOME`` if set, else ``$HOME/.local/share``.""" + override = os.environ.get("XDG_DATA_HOME") + if override: + return Path(override).expanduser() + return Path.home() / ".local" / "share" + + +def xdg_cache_home() -> Path: + """Return ``$XDG_CACHE_HOME`` if set, else ``$HOME/.cache``.""" + override = os.environ.get("XDG_CACHE_HOME") + if override: + return Path(override).expanduser() + return Path.home() / ".cache" + + +def user_data_root() -> Path: + """Return the platform-conventional data root for darnit. + + ``/darnit`` per platform (attestations + reports subtree + live under here as ``.../darnit/attestations/`` and ``.../darnit/reports/``). + """ + system = platform.system() + if system == "Darwin": + root = Path.home() / "Library" / "Application Support" / "darnit" + elif system == "Windows": + base = os.environ.get( + "LOCALAPPDATA", + str(Path.home() / "AppData" / "Local"), + ) + root = Path(base).expanduser() / "darnit" / "Data" + else: + # Linux and unknown platforms use XDG. + root = xdg_data_home() / "darnit" + logger.debug("user_data_root resolved (%s): %s", system, root) + return root + + +def user_cache_root() -> Path: + """Return the platform-conventional cache root for darnit.""" + system = platform.system() + if system == "Darwin": + root = Path.home() / "Library" / "Caches" / "darnit" + elif system == "Windows": + base = os.environ.get( + "LOCALAPPDATA", + str(Path.home() / "AppData" / "Local"), + ) + root = Path(base).expanduser() / "darnit" / "Cache" + else: + root = xdg_cache_home() / "darnit" + logger.debug("user_cache_root resolved (%s): %s", system, root) + return root diff --git a/packages/darnit/src/darnit/stores/defaults/user_local.py b/packages/darnit/src/darnit/stores/defaults/user_local.py new file mode 100644 index 00000000..ae7e45fc --- /dev/null +++ b/packages/darnit/src/darnit/stores/defaults/user_local.py @@ -0,0 +1,124 @@ +"""``user-local`` outside-repo store variants (feature 034 T025). + +Extends the ``local-fs`` backends with platform-conventional root +resolution: XDG on Linux, Apple support/cache directories on macOS, +LOCALAPPDATA on Windows. Operators write:: + + [stores.attestation] + backend = "user-local" + +and no ``root`` field is required. If they DO pass a ``root``, the +backend emits a warning and ignores it (per data-model E-002 warn-and- +ignore semantics, chosen over hard-error to reduce friction for +operators who copied a snippet from a ``local-fs`` example). + +`user-local` is deliberately NOT registered under +`darnit.stores.project` (FR-009): `.project/project.yaml` stays in the +repo. `resolve_stores` raises `StoreNotInstalled` if the operator +writes `[stores.project] backend = "user-local"`. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from darnit.core.logging import get_logger + +from .local_fs import ( + LocalFsAttestationStore, + LocalFsAuditCacheStore, + LocalFsReportStore, + _log_wrote, +) +from .platform_paths import user_cache_root, user_data_root + +logger = get_logger("stores.local") + + +def _warn_and_ignore_root(kwargs: dict[str, Any], resolved_root: Path) -> None: + """If the caller passed a `root` kwarg, log a warning; the resolved + platform root wins regardless.""" + if kwargs.get("root"): + logger.warning( + "user-local backend ignores `root = %r`; using platform default: %s", + kwargs["root"], + resolved_root, + ) + + +class UserLocalAttestationStore(LocalFsAttestationStore): + """`user-local` attestation store: root resolved via + :func:`~darnit.stores.defaults.platform_paths.user_data_root`.""" + + _BACKEND_NAME = "user-local" + + def __init__(self, **kwargs: Any) -> None: + root = user_data_root() / "attestations" + _warn_and_ignore_root(kwargs, root) + super().__init__(root=root) + + def write( + self, bundle_id: str, bundle_bytes: bytes, content_type: str + ) -> None: + # Compute target BEFORE delegating so we log with the right backend + # name even after the parent's log call runs. + target = self._target_for(bundle_id, content_type) + # Delegate to grandparent (Filesystem*Store), skipping LocalFs*'s + # own info log. We emit our own with backend="user-local". + try: + self._delegate.write(bundle_id, bundle_bytes, content_type) + except Exception as err: + from darnit.stores.errors import StoreOperationError + + raise StoreOperationError( + f"[{self._BACKEND_NAME} attestation @ {target}] {err}" + ) from err + _log_wrote("attestation", self._BACKEND_NAME, target) + + +class UserLocalReportStore(LocalFsReportStore): + """`user-local` report store.""" + + _BACKEND_NAME = "user-local" + + def __init__(self, **kwargs: Any) -> None: + root = user_data_root() / "reports" + _warn_and_ignore_root(kwargs, root) + super().__init__(root=root) + + def _write(self, report_id: str, ext: str, content: str, tag: str) -> None: + target = self._target_for(report_id, ext) + try: + self._delegate._write(report_id, ext, content) + except Exception as err: + from darnit.stores.errors import StoreOperationError + + raise StoreOperationError( + f"[{self._BACKEND_NAME} {tag} @ {target}] {err}" + ) from err + _log_wrote(tag, self._BACKEND_NAME, target) + + +class UserLocalAuditCacheStore(LocalFsAuditCacheStore): + """`user-local` audit-cache store.""" + + _BACKEND_NAME = "user-local" + + def __init__(self, **kwargs: Any) -> None: + root = user_cache_root() / "audit-cache" + _warn_and_ignore_root(kwargs, root) + super().__init__(root=root) + + def write(self, cache_key: str, envelope: dict[str, Any]) -> None: + target = self._target_for(cache_key) + self._delegate.write(cache_key, envelope) + if target.exists(): + _log_wrote(self._KIND_TAG, self._BACKEND_NAME, target) + + +__all__ = [ + "UserLocalAttestationStore", + "UserLocalAuditCacheStore", + "UserLocalReportStore", +] diff --git a/specs/034-local-output-store/checklists/requirements.md b/specs/034-local-output-store/checklists/requirements.md new file mode 100644 index 00000000..a4bfe31e --- /dev/null +++ b/specs/034-local-output-store/checklists/requirements.md @@ -0,0 +1,37 @@ +# Specification Quality Checklist: Local Output Data Store + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-09-01 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [X] No implementation details (languages, frameworks, APIs) -- backend names ("local-fs", "user-local") are user-facing config strings, not implementation choices +- [X] Focused on user value and business needs -- OSPO leader / CI operator / backward-compat use cases +- [X] Written for non-technical stakeholders -- section headings describe outcomes, not code +- [X] All mandatory sections completed + +## Requirement Completeness + +- [X] No [NEEDS CLARIFICATION] markers remain +- [X] Requirements are testable and unambiguous -- every FR states a MUST/MUST NOT with a check +- [X] Success criteria are measurable +- [X] Success criteria are technology-agnostic (no implementation details) -- SCs describe paths and file counts, not code +- [X] All acceptance scenarios are defined -- 4 user stories with Given/When/Then coverage +- [X] Edge cases are identified -- 9 edge cases enumerated +- [X] Scope is clearly bounded -- explicit Out of Scope section names 6 non-goals +- [X] Dependencies and assumptions identified -- Dependencies + Assumptions sections both filled + +## Feature Readiness + +- [X] All functional requirements have clear acceptance criteria -- FR-001..014 each mappable to a specific SC or acceptance scenario +- [X] User scenarios cover primary flows -- 4 prioritized stories including a P1 backward-compat invariant +- [X] Feature meets measurable outcomes defined in Success Criteria -- SC-001..008 close the loop +- [X] No implementation details leak into specification + +## Notes + +- Passed on first draft. Ready for `/speckit-clarify` or `/speckit-plan`. +- Constitution I (darnit-core stays filesystem-only) is respected: this feature adds NEW filesystem backends to darnit-core, not a network dependency. +- One assumption worth watching in clarify: SC-004's Windows path coverage is a stretch goal. If no Windows CI runner is available at plan time, drop the Windows integration test from tasks and keep the mocked unit test. +- Feature 033's US2 zero-config test is called out as the guarantor of SC-003; the plan phase should make it explicit that no changes to that test are permitted. diff --git a/specs/034-local-output-store/contracts/local-fs.md b/specs/034-local-output-store/contracts/local-fs.md new file mode 100644 index 00000000..44bcd37c --- /dev/null +++ b/specs/034-local-output-store/contracts/local-fs.md @@ -0,0 +1,102 @@ +# Contract: `local-fs` backend + +**Backend name (TOML)**: `local-fs` +**Registered under**: `darnit.stores.attestation`, `darnit.stores.report`, `darnit.stores.cache`, `darnit.stores.project` (see FR-009 note below) +**Config surface**: `[stores.]` block in `.baseline.toml` + +## TOML surface + +```toml +[stores.] +backend = "local-fs" +root = "" # REQUIRED. Absolute, ~-relative, or $VAR-templated. +``` + +- `root` MUST be a non-empty string. +- Any additional keys are accepted-and-ignored (extra="allow" per feature 033). + +## Root resolution (E-001, R-003) + +Applied in `__init__`, in order: + +1. `substitute_dollar_vars(root, missing="raise")` -- typo in `$VAR` name is a hard error, not a silent empty. +2. `os.path.expanduser(...)` -- `~` and `~user` expand to the running user's home directory. +3. `Path(...).resolve()` -- final absolute canonical path used for I/O and logging. + +Directory creation is deferred to first write (matches feature 033's `FilesystemAttestationStore.write`). + +## Per-kind behavior + +| Kind | Class | Delegate | +|---|---|---| +| attestation | `LocalFsAttestationStore` | `FilesystemAttestationStore(root=)` | +| report | `LocalFsReportStore` | `FilesystemReportStore(root=)` | +| cache | `LocalFsAuditCacheStore` | `FilesystemAuditCacheStore(root=)` | +| project | `LocalFsProjectStateStore` | Uses `/.project/` layout inside `root`; see below | + +### `project` kind (FR-009 note) + +`local-fs` MAY be selected for `[stores.project]`, but doing so is documented as unusual. The `.project/project.yaml` file is the CNCF `.project/` spec's canonical repo-committable artifact. Redirecting it outside the repo means downstream consumers (governance dashboards, other darnit-adjacent tools) can no longer find it via the repo path. + +If an operator explicitly configures `[stores.project] backend = "local-fs" root = ""`, the store writes `/project.yaml` and `/maintainers.yaml` (NO `.project/` subdirectory prefix -- the `` IS the `.project/` equivalent). Documentation in `docs/plugin-authoring/stores.md` covers this. + +`user-local` is deliberately not registered for `[stores.project]`. + +## Write contract (per Protocol) + +Unchanged from feature 033's `Filesystem*Store`. Every method delegates to the underlying `Filesystem*Store` after the info-log line (see below). + +- `AttestationStore.write(bundle_id, bundle_bytes, content_type)`: + - Path: `/` where `` maps from `content_type` via the existing `_CONTENT_TYPE_TO_EXT` table (`in-toto+json` -> `.intoto.json`, `sigstore.bundle+json` -> `.sigstore.json`, else `.bin`). + - Errors: OSError, PermissionError propagate to the audit driver -> `StoreOperationError`. +- `ReportStore.write_markdown(report_id, contents)` / `.write_json` / `.write_sarif`: + - Paths: `/.md` / `.json` / `.sarif`. + - Errors: same propagation. +- `AuditCacheStore.read(cache_key) -> dict | None`: returns None on miss, corrupt JSON, or OSError (feature 033 FR-011: best-effort). +- `AuditCacheStore.write(cache_key, envelope)`: tempfile-then-rename (same directory as target, cross-fs safe). Best-effort -- swallows all exceptions to a debug log. +- `close()`: no-op. + +## Sanitization (SC-005) + +All identifiers passed as filename components (`bundle_id`, `report_id`, `cache_key`) go through the shared `_sanitize_filename` regex from `stores/defaults/attestation.py`: + +``` +_FILENAME_UNSAFE = re.compile(r"[^A-Za-z0-9._+@-]") +_sanitize_filename(x) = _FILENAME_UNSAFE.sub("_", x) or "unnamed" +``` + +Consequence: `bundle_id = "../../etc/foo"` produces a filename `.._.._etc_foo.intoto.json` INSIDE ``. Path traversal is impossible by construction. + +## Logging (FR-015 / SC-009) + +Every successful write emits one info-level log line to logger `darnit.stores.local`: + +``` +INFO wrote (local-fs): +``` + +- ``: `"attestation"`, `"report:markdown"` | `"report:json"` | `"report:sarif"`, or `"cache"`. +- ``: the full absolute path after sanitization and content-type-to-extension mapping. This is the exact path the file lives at. + +Failed writes do NOT emit this line (the OSError propagation is the failure signal). Cache best-effort failures emit a debug-level line only (already existing). + +## Error modes (recap of feature 033 Protocol contracts) + +| Protocol | Error condition | Behavior | +|---|---|---| +| `AttestationStore` | Directory unwritable, disk full | Raises `OSError` -> `StoreOperationError` -> operator sees clear message naming backend + kind + path (SC-008) | +| `ReportStore` | Same | Same | +| `AuditCacheStore` | Same | Best-effort: swallow to debug log; read returns None; audit continues | +| `ProjectStateStore` | Same on read/write | Propagates; affected controls resolve WARN | + +## Test surface + +- `tests/darnit/stores/test_local_fs_backend.py`: per-kind write round-trip, `$VAR` interpolation (present + missing = raise), `~` expansion, path-traversal sanitization (SC-005), cross-filesystem `root` (uses `tmp_path` on the test's temp filesystem which may differ from `/tmp`). +- `tests/darnit/stores/test_local_fs_logging.py`: caplog assertion on each write (SC-009), zero-log assertion for `Filesystem*Store` writes. +- `tests/darnit/stores/test_local_fs_isolation.py`: `[stores.attestation] backend = "local-fs"` + others unset -> only attestation redirects (SC-007). + +## Non-scope + +- No retention/pruning inside `root`. Operator manages it. +- No cross-filesystem atomic-write for `AttestationStore`/`ReportStore` (only `AuditCacheStore` needs it, and R-004 confirms it's handled). +- No encryption at rest. diff --git a/specs/034-local-output-store/contracts/user-local.md b/specs/034-local-output-store/contracts/user-local.md new file mode 100644 index 00000000..60402693 --- /dev/null +++ b/specs/034-local-output-store/contracts/user-local.md @@ -0,0 +1,90 @@ +# Contract: `user-local` backend + +**Backend name (TOML)**: `user-local` +**Registered under**: `darnit.stores.attestation`, `darnit.stores.report`, `darnit.stores.cache` +**NOT registered under**: `darnit.stores.project` (FR-009: `.project/` stays in-repo) +**Config surface**: `[stores.]` block in `.baseline.toml` + +## TOML surface + +```toml +[stores.] +backend = "user-local" +# No config keys are required or honored. Any `root = "..."` is ignored with a warning. +``` + +- No config knobs. The whole point of `user-local` is that the operator doesn't spell out paths. + +## Root resolution + +Computed at `__init__` from the platform via `platform_paths` (E-003): + +| Platform | Data root (attestations, reports) | Cache root (audit cache) | +|---|---|---| +| Linux | `${XDG_DATA_HOME:-$HOME/.local/share}/darnit/` | `${XDG_CACHE_HOME:-$HOME/.cache}/darnit/` | +| macOS | `~/Library/Application Support/darnit/` | `~/Library/Caches/darnit/` | +| Windows | `%LOCALAPPDATA%\darnit\Data\` | `%LOCALAPPDATA%\darnit\Cache\` | +| Unknown platform | XDG fallback (same as Linux) | XDG fallback (same as Linux) | + +Per artifact kind, the store appends its own subdirectory: + +| Kind | Class | Resolved path | +|---|---|---| +| attestation | `UserLocalAttestationStore` | `/attestations/` | +| report | `UserLocalReportStore` | `/reports/` | +| cache | `UserLocalAuditCacheStore` | `/audit-cache/` | + +## Extra `root` kwarg: warn-and-ignore (FR-004) + +Per FR-004: "Passing `root` to `user-local` explicitly MUST either be ignored with a warning or rejected with a clear error; the plan phase picks between the two." **Decision: warn-and-ignore**, matching E-002. + +Behavior: + +- On `__init__`, if the incoming kwargs contain a non-empty `root` value, emit ONE warning-level log line to logger `darnit.stores.local`: + ``` + WARNING user-local backend ignores `root = `; using platform default: + ``` +- Then proceed with the platform-computed root as normal. The extraneous kwarg is dropped. + +Rationale: less disruptive than a hard error for operators who copied a config example from a `local-fs` block. The warning is loud enough to correlate to the offending config line if the operator is watching logs. + +## Delegation to `local-fs` + +`UserLocal*Store` inherits from `LocalFs*Store`. Once platform resolution is done, `super().__init__(root=)` runs the same chain: `_sanitize_filename`, delegate to `Filesystem*Store` for I/O, tempfile-then-rename for cache. + +## Logging (FR-015 / SC-009) + +Every successful write emits one info-level log line to logger `darnit.stores.local`: + +``` +INFO wrote (user-local): +``` + +- ``: same values as `local-fs` -- `"attestation"`, `"report:markdown"` | `"report:json"` | `"report:sarif"`, or `"cache"`. +- ``: the full absolute path computed via `platform_paths` + subdirectory. + +## `_StoreBundle` lazy-instantiation guarantee (R-005) + +`UserLocal*Store.__init__` reads `platform.system()`, environment variables (`$XDG_DATA_HOME` / `$LOCALAPPDATA`), and `Path.home()`. This work runs ONLY when the store is first accessed via a `_StoreBundle` property -- feature 033's factory closures defer construction. An audit that never touches attestations pays zero cost even if `[stores.attestation] backend = "user-local"` is configured. + +## Test surface + +- `tests/darnit/stores/test_user_local_backend.py`: + - parametric per-kind write round-trip on the runtime platform; + - explicit-root warn-and-ignore assertion; + - assertion that `super().__init__(root=)` is called with the expected computed path. +- `tests/darnit/stores/test_platform_paths.py`: + - `xdg_data_home()` with `$XDG_DATA_HOME` set + unset; + - `xdg_cache_home()` with `$XDG_CACHE_HOME` set + unset; + - `user_data_root()` / `user_cache_root()` with `platform.system()` monkeypatched to `"Linux"`, `"Darwin"`, `"Windows"`, and `"FreeBSD"` (unknown fallback). +- `tests/darnit/stores/test_local_fs_isolation.py`: with `[stores.attestation] backend = "user-local"` + `[stores.project]` unset, `/.project/` is still used for project state. + +## Interaction with `.project/` (FR-009) + +There is no `UserLocalProjectStateStore` class. There is no entry-point registration under `darnit.stores.project`. An operator writing `[stores.project] backend = "user-local"` in TOML MUST get a `StoreNotInstalled` error at `resolve_stores()` time -- feature 033 raises before any control runs, satisfying SC-007's stronger interpretation ("`.project/` stays in-repo when other kinds are user-local"). + +## Non-scope + +- No `root` config surface (the whole point of `user-local`). +- No `.project/` support. +- No mixed-mode (e.g., "user-local for data, XDG override for cache") -- the operator uses two separate `[stores.]` blocks if they want asymmetric config, one `user-local` and one `local-fs`. diff --git a/specs/034-local-output-store/data-model.md b/specs/034-local-output-store/data-model.md new file mode 100644 index 00000000..89f28eca --- /dev/null +++ b/specs/034-local-output-store/data-model.md @@ -0,0 +1,161 @@ +# Data Model: Local Output Data Store (Phase 1) + +**Feature**: 034-local-output-store +**Date**: 2026-09-01 + +Five entities. No new persisted data schema; every entity is either a Python class hierarchy or a TOML config field. + +--- + +## E-001: `LocalFs*Store` (3 classes + 1 optional) + +**Purpose**: filesystem-backed store variants that write to any configurable `root` path outside the audited repository. + +**Classes**: + +| Class | Protocol satisfied | File | +|---|---|---| +| `LocalFsAttestationStore` | `AttestationStore` | `packages/darnit/src/darnit/stores/defaults/local_fs.py` | +| `LocalFsReportStore` | `ReportStore` | same file | +| `LocalFsAuditCacheStore` | `AuditCacheStore` | same file | +| `LocalFsProjectStateStore` | `ProjectStateStore` | same file (optional -- registered but flagged unusual per FR-009) | + +**Construction contract** (shared shape, per class): + +- `__init__(self, root: str | Path, **_)`. + - `root: str` accepted from TOML with `$VAR` interpolation and `~` expansion applied at construction, in that order. + - `root: Path` accepted from Python callers (tests) verbatim. + - Extra `**_` kwargs are accepted-and-ignored to preserve compatibility with feature 033's `_instantiate_plugin` kwargs pass-through. +- Delegates to the matching `Filesystem*Store(root=)` internally. + +**Root resolution rules** (applied in order at `__init__`): + +1. If input is a `Path`, use as-is (test-only shortcut). +2. If input is a `str`, run `substitute_dollar_vars(root, missing="raise")` (per R-003). A missing env var raises `KeyError` here, before the audit begins. +3. Run `os.path.expanduser(resolved)` to expand `~`. +4. Convert to absolute `Path` via `Path(resolved).resolve()` for logging + downstream I/O. +5. Do NOT create the directory yet -- creation happens on the first write, consistent with `FilesystemAttestationStore`. + +**Write contract**: delegate verbatim to the corresponding `Filesystem*Store` write method. The delegate handles directory creation, filename sanitization (`_sanitize_filename`), tempfile-then-rename for cache, and content-type-to-extension mapping for attestation. + +**Logging obligation** (FR-015): after every successful write, emit exactly one info-level line to logger `darnit.stores.local`: + +``` +INFO wrote (local-fs): +``` + +Where `` is `"attestation"`, `"report:markdown"` / `"report:json"` / `"report:sarif"`, or `"cache"`. The `report:*` split gives the operator one log line per format so multi-format audit runs are self-documenting. + +**Close contract**: `close()` is a no-op, matching the delegate. + +--- + +## E-002: `UserLocal*Store` (3 classes) + +**Purpose**: convenience variants that resolve `root` from platform conventions so operators don't spell out full paths. + +**Classes**: + +| Class | Protocol satisfied | Data root computation | Cache root computation | +|---|---|---|---| +| `UserLocalAttestationStore` | `AttestationStore` | `platform_paths.user_data_root() / "attestations"` | N/A | +| `UserLocalReportStore` | `ReportStore` | `platform_paths.user_data_root() / "reports"` | N/A | +| `UserLocalAuditCacheStore` | `AuditCacheStore` | N/A | `platform_paths.user_cache_root() / "audit-cache"` | + +**Not defined**: `UserLocalProjectStateStore` -- FR-009 requires `.project/` stay in-repo. No entry-point registration for `stores.project` at `user-local`. + +**Construction contract**: + +- `__init__(self, **kwargs)`. +- Accepts and ignores any `root` kwarg the caller might pass (per FR-004: "MUST either be ignored with a warning or rejected with a clear error"). Chosen behavior: **log a warning at info level** and proceed with the platform-computed root. Rationale: less disruptive than an error for operators who accidentally copied a `root` from a `local-fs` example; the warning names the resolved platform path so they can correlate. +- Extra `**kwargs` are otherwise accepted-and-ignored. + +**Root resolution**: computed once in `__init__` via a call into `platform_paths` (see E-003). The result is passed to `super().__init__(root=)` (extends `LocalFs*Store`), so all downstream logic (sanitizer, delegation, logging) is inherited. + +**Logging obligation** (FR-015): same format as E-001, with `` = `"user-local"`. + +--- + +## E-003: `platform_paths` module + +**Purpose**: OS-dispatching path resolution for `user-local`. + +**Public API**: + +```python +def xdg_data_home() -> Path: ... # Linux: $XDG_DATA_HOME or ~/.local/share +def xdg_cache_home() -> Path: ... # Linux: $XDG_CACHE_HOME or ~/.cache +def user_data_root() -> Path: ... # returns /darnit +def user_cache_root() -> Path: ... # returns /darnit +``` + +`user_data_root()` and `user_cache_root()` are the only two entry points `UserLocal*Store` calls; `xdg_data_home()` / `xdg_cache_home()` are exported for direct testing. + +**Platform dispatch** (inside `user_data_root` / `user_cache_root`): + +```python +system = platform.system() # "Linux" | "Darwin" | "Windows" + +# data root +if system == "Darwin": + return Path.home() / "Library" / "Application Support" / "darnit" +if system == "Windows": + return Path(os.environ.get("LOCALAPPDATA", str(Path.home() / "AppData" / "Local"))) / "darnit" / "Data" +# Linux and unknown: XDG +return xdg_data_home() / "darnit" +``` + +Analogous for cache root, substituting `Library/Caches`, `LOCALAPPDATA\...\Cache`, and `xdg_cache_home()`. + +**Unknown platform** (`platform.system()` returns e.g. `"FreeBSD"`): falls through to the XDG branch (data at `$XDG_DATA_HOME` or `~/.local/share/darnit`, cache at `$XDG_CACHE_HOME` or `~/.cache/darnit`). Documented in the module docstring. + +--- + +## E-004: `root` config field + +**Purpose**: the TOML surface. Applies to `local-fs` only. `user-local` reads no config; if the operator writes `root = "..."` on a `[stores.] backend = "user-local"` block, `UserLocal*Store` logs a warning and ignores it. + +**TOML surface**: + +```toml +[stores.attestation] +backend = "local-fs" +root = "/absolute/path" # OR +root = "~/subpath" # OR +root = "$VAR/subpath" # OR +root = "~/$VAR/subpath" # combinations OK; $VAR resolved before ~ +``` + +**Schema-level**: no schema change. Feature 033's `StoreBlock` uses `extra="allow"`, so arbitrary backend-specific keys (like `root`) pass through the config layer verbatim into `_instantiate_plugin`'s kwargs. + +**Interpolation order** (per R-003 + E-001): + +1. `substitute_dollar_vars(root, missing="raise")` +2. `os.path.expanduser(...)` +3. `Path(...).resolve()` + +Steps 1 and 2 both run even if the input contains neither `$` nor `~` (both are no-ops in that case, cheap). + +**Absent `root` on `local-fs`**: caller receives `TypeError` from Python (missing required kwarg) -- surfaces to the operator as "TypeError: LocalFsAttestationStore.__init__() missing 1 required argument: 'root'". Feature 033's `_instantiate_plugin` propagates the exception into `StoreOperationError` at the audit boundary, which is the loudest possible signal. + +--- + +## E-005: Info-log format for outside-repo writes + +**Purpose**: single, greppable line per write. Satisfies FR-015 and enables SC-009's caplog assertion. + +**Log record**: + +- Logger name: `darnit.stores.local` (shared across `local-fs` and `user-local`) +- Level: INFO +- Message template: `"wrote %s (%s): %s"` with args `(kind, backend, str(resolved_path))` + +Example emitted line: + +``` +INFO darnit.stores.local: wrote attestation (local-fs): /home/mike/darnit-attestations/acme-widget-baseline-attestation.intoto.json +``` + +**Zero-config exemption**: `Filesystem*Store` classes are unchanged. They do NOT emit this line. Zero-config audits produce zero log lines from this logger, which is what SC-009 requires. + +**Multi-format reports**: `LocalFsReportStore.write_markdown` / `.write_json` / `.write_sarif` each emit one line with `kind = "report:markdown"` / `"report:json"` / `"report:sarif"`. Three-format audit runs yield three log lines. diff --git a/specs/034-local-output-store/plan.md b/specs/034-local-output-store/plan.md new file mode 100644 index 00000000..2c48bd72 --- /dev/null +++ b/specs/034-local-output-store/plan.md @@ -0,0 +1,157 @@ +# Implementation Plan: Local Output Data Store + +**Branch**: `034-local-output-store` | **Date**: 2026-09-01 | **Spec**: [spec.md](spec.md) + +**Input**: Feature specification from `/specs/034-local-output-store/spec.md` + +## Summary + +Two new filesystem-backed `Store` backends, `local-fs` and `user-local`, that let operators write attestations / reports / audit-cache OUTSIDE the audited repository via `.baseline.toml`'s `[stores.] backend = "..."` selector. Extends feature 033's plugin surface with no Protocol changes; every backend ships inside `packages/darnit/src/darnit/stores/defaults/` alongside the existing in-repo `Filesystem*Store` defaults. Zero-config audits are byte-for-byte unchanged. + +Technical approach in three moves: + +1. **`LocalFsAttestationStore` / `LocalFsReportStore` / `LocalFsAuditCacheStore`** wrap the existing `Filesystem*Store` classes and swap the default in-repo root for a config-driven `root` (absolute path, `~`-expanded, or `$VAR`-substituted via `darnit.core.env_subst`). +2. **`UserLocalAttestationStore` / `UserLocalReportStore` / `UserLocalAuditCacheStore`** compute a platform-appropriate root at construction time (XDG on Linux, `~/Library/{Application Support,Caches}/darnit` on macOS, `%LOCALAPPDATA%\darnit\{Data,Cache}` on Windows) and delegate to the `LocalFs*` layer. +3. **Entry-point registrations** under `darnit.stores.{attestation,report,cache}` for both backend names. `.project/` gets `local-fs` too for completeness, but `user-local` is deliberately NOT registered for `stores.project` (per FR-009: `.project/` stays in-repo). + +Every outside-repo write emits one info-level log line per artifact naming the backend, kind, and resolved path (FR-015 / SC-009). All feature 033 constitutional guarantees hold: no new runtime dep, no framework-side wiring changes, no `Store` Protocol methods added. + +## Technical Context + +**Language/Version**: Python 3.11 / 3.12 (workspace targets from CLAUDE.md) + +**Primary Dependencies**: stdlib only. `pathlib.Path`, `os.path.expanduser`, `os.environ` for `$VAR` (already fronted by `darnit.core.env_subst` from feature 033). No new packages. + +**Storage**: Filesystem. Local. Outside the audited repo when configured; per-repo `.darnit/` when not. + +**Testing**: pytest (workspace default). Existing feature-033 test surface at `tests/darnit/stores/` is the template for the new tests. Some tests parametrize by platform. + +**Target Platform**: macOS + Linux fully supported. Windows is stretch goal per spec assumption; unit-tested with a mocked platform-name lookup, integration-tested only if a Windows CI runner exists at implement time. + +**Project Type**: Library extension inside an existing workspace member (`packages/darnit/`). No new package. + +**Performance Goals**: N/A. Filesystem I/O bounded by disk speed. One additional info-log per artifact per audit is trivial overhead. + +**Constraints**: +- No new runtime dependency (FR-014). +- No changes to `Store` Protocol methods. +- `.project/` MUST stay in-repo even when `user-local` is selected for other kinds (FR-009). +- Feature 033's `test_us2_zero_config.py` MUST continue to pass unchanged (SC-003). +- Path-traversal sanitizer reused from `packages/darnit/src/darnit/stores/defaults/` (spec Key Entities note). + +**Scale/Scope**: 3 artifact kinds (attestation, report, cache) x 2 backends = 6 concrete classes + up to 7 entry-point registrations. Estimated ~150 lines of implementation code, ~250 lines of tests, ~1 docs section. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +| Principle | Status | Notes | +|---|---|---| +| I. Plugin Separation | PASS | New backends live in `packages/darnit/src/darnit/stores/defaults/` inside darnit-core. Constitution I forbids darnit-core importing implementation packages; adding built-in filesystem behaviors is explicitly compatible (matches feature 033's precedent). | +| II. Conservative-by-Default | PASS | FR-013 preserves feature 033's per-Protocol failure semantics: no silent fallback (SC-008), attestation write errors surface, cache is best-effort. No new "compliant" claim path opens up. | +| III. TOML-First Architecture | PASS | Everything configured through `[stores.] backend = "..." root = "..."` in TOML. Zero Python for operators. | +| IV. Never Guess User Values | N/A | Storage backends don't produce or consume user-judgment values. `.project/` stays in-repo (FR-009) so no user-judgment value is silently relocated. | +| V. Sieve Pipeline Integrity | N/A | Backends are not sieve passes; they don't participate in the 4-phase pipeline. | + +**Initial gate: PASS.** No violations. Re-check after Phase 1 design. + +## Project Structure + +### Documentation (this feature) + +```text +specs/034-local-output-store/ +├── plan.md # this file (/speckit-plan output) +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +├── contracts/ # Phase 1 output +│ ├── local-fs.md +│ └── user-local.md +└── tasks.md # Phase 2 output (/speckit-tasks) +``` + +### Source Code (repository root) + +```text +packages/darnit/src/darnit/stores/defaults/ +├── __init__.py # add new re-exports +├── attestation.py # FilesystemAttestationStore (existing) -- untouched +├── cache.py # FilesystemAuditCacheStore (existing) -- untouched +├── project.py # FilesystemProjectStateStore (existing) -- untouched +├── report.py # FilesystemReportStore (existing) -- untouched +├── local_fs.py # NEW: LocalFs{Attestation,Report,Cache,Project}Store +├── user_local.py # NEW: UserLocal{Attestation,Report,Cache}Store +└── platform_paths.py # NEW: Linux/macOS/Windows XDG-style path resolution + +packages/darnit/pyproject.toml # ADD up to 7 entry-point registrations under + # darnit.stores.{project,attestation,report,cache} + # (project gets local-fs only, not user-local; FR-009) + +tests/darnit/stores/ +├── test_local_fs_backend.py # NEW: parametric per-kind coverage of local-fs +├── test_user_local_backend.py # NEW: same, plus platform-parameterized root resolution +├── test_platform_paths.py # NEW: unit tests for the resolver +├── test_local_fs_logging.py # NEW: caplog assertions for FR-015 / SC-009 +├── test_local_fs_isolation.py # NEW: attest -> local-fs, others stay in-repo (SC-007) +└── (existing tests unchanged; + test_us2_zero_config.py is the SC-003 witness) + +docs/plugin-authoring/ +└── stores.md # ADD a "Writing artifacts outside the repo" section + # (SC-006) +``` + +**Structure decision**: single-package extension inside `packages/darnit/`. No new workspace member, no new test dir. The two new backend files and one platform-paths helper each live next to their existing siblings under `stores/defaults/`, matching the shape feature 033 established. Tests mirror the module split so failures point at one file. + +## Phase 0: Outline & Research + +No NEEDS CLARIFICATION markers survived the clarify pass (spec's Clarifications section, session 2026-09-01). The five items in `research.md` are all dependencies already-existing in the feature 033 surface; research consolidates the API contracts and edge cases: + +1. What platform-path conventions should `user-local` follow, cross-referenced against `platformdirs`-style community norms without introducing that dependency? +2. What exact filename sanitizer does feature 033's `FilesystemAttestationStore` use, and can `LocalFs*` reuse it verbatim? +3. What does the existing `darnit.core.env_subst` interface look like -- is `missing="leave"` the right mode for `root`, or should we prefer `"raise"` to fail fast on typos? +4. What tempfile-then-rename semantics does `FilesystemAuditCacheStore` use, and do they hold when `root` is on a different filesystem than the system tempdir? +5. Does feature 033's `_StoreBundle` lazy-instantiation still work correctly when a `user-local` backend's `__init__` does platform-path resolution? + +All five resolvable from source. See `research.md` for the resolutions. + +## Phase 1: Design & Contracts + +### Data model + +See [data-model.md](data-model.md). Five entities: + +- **`LocalFs*Store`**: three concrete classes (attestation, report, cache) that wrap the existing `Filesystem*Store` with a `root` computed from config. A fourth `LocalFsProjectStateStore` is defined but its use is documented as unusual. +- **`UserLocal*Store`**: three concrete classes (attestation, report, cache) that resolve `root` from platform conventions and delegate to `LocalFs*Store`. +- **`platform_paths`**: functions `xdg_data_home()`, `xdg_cache_home()`, `user_data_root()`, `user_cache_root()` -- OS-dispatching path resolution. +- **`root` config field**: TOML string on the `[stores.]` block. Absolute, `~`-relative, or `$VAR`-templated. Interpolated at store construction, not lazily on write. +- **Info-log format**: single `logger.info("wrote %s (%s): %s", kind, backend, resolved_path)` per artifact write. Shared logger name `darnit.stores.local`. + +### Contracts + +See `contracts/local-fs.md` and `contracts/user-local.md`. Each contract enumerates the config keys accepted, the resolved-root computation, error modes (per Protocol), logging obligations, and the interaction with `.project/` (`local-fs` may be selected for `stores.project`; `user-local` is NOT registered for `stores.project`). + +### Quickstart + +See [quickstart.md](quickstart.md). Three worked examples: + +1. **Consolidate attestations for an OSPO leader**: `[stores.attestation] backend = "local-fs" root = "$DARNIT_ATT_ROOT"` + explanation of env-var-driven multi-repo templating (the Q1 answer). +2. **CI runner cache + reports**: two `[stores.]` blocks pointing at runner-provided paths. +3. **XDG defaults on Linux / macOS conventions**: `backend = "user-local"` per kind; explain the resolved roots per platform. + +### Agent context update + +CLAUDE.md's `` marker currently points at feature 033's plan. Update to point at this feature's plan (done at end of Phase 1). + +## Constitution re-check (post-design) + +| Principle | Status | +|---|---| +| I. Plugin Separation | PASS -- design only touches `packages/darnit/src/darnit/stores/defaults/`, adds no cross-package imports | +| II. Conservative-by-Default | PASS -- per-Protocol failure semantics preserved verbatim; no new "compliant" claim path opens up | +| III. TOML-First Architecture | PASS -- config surface stays TOML; backend names are first-class selectors | +| IV. Never Guess User Values | N/A | +| V. Sieve Pipeline Integrity | N/A | + +**Final gate: PASS.** Ready for `/speckit-tasks`. diff --git a/specs/034-local-output-store/quickstart.md b/specs/034-local-output-store/quickstart.md new file mode 100644 index 00000000..0d35306a --- /dev/null +++ b/specs/034-local-output-store/quickstart.md @@ -0,0 +1,146 @@ +# Quickstart: Writing artifacts outside the audited repo (034) + +**Feature**: 034-local-output-store +**Audience**: operators configuring darnit for outside-repo storage + +Three end-to-end examples, in order of adoption difficulty. Each is a two-block `.baseline.toml` snippet with no further code changes. + +--- + +## 1. OSPO leader consolidates attestations across many repos + +**Goal**: every audit's attestation lands in `~/darnit-attestations/`, backed up together, discoverable to your SBOM pipeline. Not touching the audited repos. + +**Config to add to each repo's `.baseline.toml`**: + +```toml +[stores.attestation] +backend = "local-fs" +root = "$DARNIT_ATT_ROOT" # or a literal path like "~/darnit-attestations" +``` + +**Multi-repo templating**: darnit does NOT ship an org-level config layer (see spec Q1). The operator handles "one config for 30 repos" via one of: + +1. **Env-var interpolation (easiest)**: keep `root = "$DARNIT_ATT_ROOT"` in every repo's `.baseline.toml`. Set `DARNIT_ATT_ROOT=~/darnit-attestations` once per machine (shell profile, systemd unit, CI runner config). All 30 repos share that root without duplicating literals. +2. **CI/CD templating**: your CI workflow rewrites `.baseline.toml` before invoking `darnit audit` (envsubst, sed, Jinja). +3. **Cookiecutter / repo-init tool**: one-shot copy of the `[stores.attestation]` block into each repo the first time you onboard it. + +**Verify**: + +```bash +cd my-repo +darnit audit . +# audit runs; attestation lands at $DARNIT_ATT_ROOT/-baseline-attestation.intoto.json +ls -1 $DARNIT_ATT_ROOT/ +# should list the newly-written bundle +grep "wrote attestation (local-fs)" audit-log +# should show the resolved absolute path +``` + +**Log line** (FR-015): + +``` +INFO darnit.stores.local: wrote attestation (local-fs): /home/mike/darnit-attestations/acme-widget-baseline-attestation.intoto.json +``` + +**What did NOT change**: `/.darnit/attestations/` is not created (SC-002). Reports and audit-cache still live in-repo (SC-007). `.project/project.yaml` still lives in-repo (FR-009). + +--- + +## 2. CI runner: cache in a persistent volume, reports as job artifacts + +**Goal**: on a CI runner where the repo is a fresh checkout per job, keep the audit cache in a persistent volume the CI system already caches between jobs, and drop reports into a per-job artifacts directory the CI system already uploads. + +**Config**: + +```toml +[stores.cache] +backend = "local-fs" +root = "$RUNNER_CACHE_DIR/darnit" + +[stores.report] +backend = "local-fs" +root = "$RUNNER_ARTIFACTS_DIR/darnit-reports" +``` + +**Behavior**: + +- First-run: cache write goes to `$RUNNER_CACHE_DIR/darnit/`. Next-run: cache read hits because the runner restored the cache; audit skips the sieve loop, cost drops to seconds. +- Reports land where the CI runner's artifact-upload step is already configured to look. Markdown/JSON/SARIF each become their own file under that root; one log line per format. +- Attestations still land in-repo (unless a third block redirects them). + +**Verify**: + +```bash +# Run twice back-to-back, second should hit cache: +darnit audit . # miss, populates $RUNNER_CACHE_DIR/darnit/.json +darnit audit . # hit, log shows cache read +ls $RUNNER_ARTIFACTS_DIR/darnit-reports/ +# should list .md, .json, .sarif +``` + +**Log lines**: + +``` +INFO darnit.stores.local: wrote report:markdown (local-fs): /ci/artifacts/darnit-reports/audit-2026-09-01.md +INFO darnit.stores.local: wrote report:json (local-fs): /ci/artifacts/darnit-reports/audit-2026-09-01.json +INFO darnit.stores.local: wrote report:sarif (local-fs): /ci/artifacts/darnit-reports/audit-2026-09-01.sarif +``` + +--- + +## 3. Individual developer: XDG defaults on Linux, Apple conventions on macOS + +**Goal**: no path-typing at all. Let darnit put artifacts where your OS says user-scoped app data goes. + +**Config**: + +```toml +[stores.attestation] +backend = "user-local" + +[stores.report] +backend = "user-local" + +[stores.cache] +backend = "user-local" +``` + +**Resolved paths**: + +| Platform | Attestations | Reports | Cache | +|---|---|---|---| +| Linux (XDG defaults) | `~/.local/share/darnit/attestations/` | `~/.local/share/darnit/reports/` | `~/.cache/darnit/audit-cache/` | +| Linux (`XDG_DATA_HOME=/mnt/x`) | `/mnt/x/darnit/attestations/` | `/mnt/x/darnit/reports/` | (uses `XDG_CACHE_HOME` similarly) | +| macOS | `~/Library/Application Support/darnit/attestations/` | `~/Library/Application Support/darnit/reports/` | `~/Library/Caches/darnit/audit-cache/` | +| Windows | `%LOCALAPPDATA%\darnit\Data\attestations\` | `%LOCALAPPDATA%\darnit\Data\reports\` | `%LOCALAPPDATA%\darnit\Cache\audit-cache\` | + +**Verify** (macOS): + +```bash +darnit audit . +ls ~/Library/Application\ Support/darnit/attestations/ +ls ~/Library/Caches/darnit/audit-cache/ +``` + +**Log line**: + +``` +INFO darnit.stores.local: wrote attestation (user-local): /Users/mike/Library/Application Support/darnit/attestations/my-repo-baseline-attestation.intoto.json +``` + +**What did NOT change**: `.project/project.yaml` is not redirected -- there is no `user-local` registration for `[stores.project]`. Even if the operator sets `[stores.project] backend = "user-local"` explicitly in TOML, `darnit audit` fails at `resolve_stores()` time with `StoreNotInstalled: user-local not registered under darnit.stores.project` before any control runs. (This is FR-009 in enforced form.) + +--- + +## Troubleshooting + +**`KeyError: DARNIT_ATT_ROOT`** at audit start: the env var referenced in `root` isn't set. `local-fs` uses `missing="raise"` mode on env-var interpolation (research R-003), so a typo or an unset variable is a hard error. Fix by exporting the variable OR by writing a literal path. + +**`StoreOperationError: attestation write failed`**: the resolved `root` is unwritable, the disk is full, or the file already exists and is locked. The error names the backend, kind, and path (SC-008). darnit does NOT silently fall back to the in-repo default (feature 033 FR-012). + +**Cache misses on second run**: check that the same `$RUNNER_CACHE_DIR` was restored between runs and that the git HEAD commit hash matches. Cache TTL is 3600s by default; refresh the cache if the audit is older. + +**File landed with weird `_`s in the name**: your `bundle_id` / `report_id` / `cache_key` contained filesystem-unsafe characters (`/`, `..`, shell metacharacters, spaces). The store sanitized them to prevent path traversal (SC-005). This is expected. Rename the caller's identifier if you want cleaner filenames. + +**`user-local` chose the wrong platform**: if you're running on a container image whose `platform.system()` returns something unexpected, `user-local` falls back to XDG defaults. Force a specific path by switching to `local-fs` with an explicit `root`. diff --git a/specs/034-local-output-store/research.md b/specs/034-local-output-store/research.md new file mode 100644 index 00000000..8978eefe --- /dev/null +++ b/specs/034-local-output-store/research.md @@ -0,0 +1,126 @@ +# Research: Local Output Data Store (Phase 0) + +**Feature**: 034-local-output-store +**Date**: 2026-09-01 + +All items resolvable from source. No NEEDS CLARIFICATION markers. + +--- + +## R-001: Platform-path conventions for `user-local` + +**Decision**: hand-roll three platform-specific resolvers inside `platform_paths.py`, matching the XDG spec on Linux, Apple support/cache conventions on macOS, and `LOCALAPPDATA` on Windows. Do NOT introduce `platformdirs`. + +**Resolutions per platform**: + +| Kind | Linux | macOS | Windows | +|---|---|---|---| +| data (attestations, reports) | `${XDG_DATA_HOME:-$HOME/.local/share}/darnit/` | `~/Library/Application Support/darnit/` | `%LOCALAPPDATA%\darnit\Data\` | +| cache (audit cache) | `${XDG_CACHE_HOME:-$HOME/.cache}/darnit/` | `~/Library/Caches/darnit/` | `%LOCALAPPDATA%\darnit\Cache\` | + +Per artifact kind: `/attestations/`, `/reports/`, `/audit-cache/`. + +**Rationale**: The XDG defaults and Apple conventions are stable enough that hand-rolling three ~5-line resolvers is cheaper than pulling in `platformdirs` (which would violate FR-014's no-new-runtime-dependency constraint). The three implementations fit in ~40 lines total including the OS-dispatch. `platform.system()` returns `"Linux"` / `"Darwin"` / `"Windows"`; unknown platforms fall through to the Linux XDG path as the safest heuristic. + +**Alternatives considered**: +- Introduce `platformdirs`. Cleaner code but violates FR-014. +- Use `pathlib.Path.home() / ".darnit" / ` uniformly. Simpler but ignores Windows LOCALAPPDATA convention (`~/.darnit` on Windows is a home-dir dotfile, not the OS-idiomatic location). + +--- + +## R-002: Filename sanitizer reuse + +**Decision**: Import `_sanitize_filename` from `darnit.stores.defaults.attestation` in `local_fs.py`. Do NOT copy-paste. `FilesystemAuditCacheStore` (`stores/defaults/cache.py:22`) already sets this precedent. + +**Source of truth** (`stores/defaults/attestation.py:48-53`): + +```python +_FILENAME_UNSAFE = re.compile(r"[^A-Za-z0-9._+@-]") + +def _sanitize_filename(name: str) -> str: + """Replace filesystem-unsafe characters with `_` for cross-platform safety.""" + return _FILENAME_UNSAFE.sub("_", name) or "unnamed" +``` + +**Rationale**: Regex handles path-traversal sanitization (SC-005) at the leaf-name level -- `/`, `..`, and shell metachars all map to `_`. Empty-string safety via the `or "unnamed"` fallback. Since attestation, report, and cache all pass their identifier through this function today, the new backends inherit the property automatically when they delegate to the existing classes. + +**Consequence for path-traversal test (SC-005)**: `bundle_id = "../../etc/foo"` will produce a filename like `.._.._etc_foo.intoto.json` (each of `/` and space-adjacent chars replaced with `_`). Test asserts on the sanitized filename directly, not on a "no directory traversal escaped" absence check, because the file cannot escape by construction. + +--- + +## R-003: `env_subst` mode for `root` interpolation + +**Decision**: Use `missing="raise"` (fail-fast) for `root`, matching the operator-facing failure surface the spec wants for typos. + +**API** (`packages/darnit/src/darnit/core/env_subst.py`): + +```python +MissingMode = Literal["empty", "raise", "leave"] + +def substitute_dollar_vars( + template: str, + env: Mapping[str, str] | None = None, + *, + missing: MissingMode = "empty", +) -> str: ... +``` + +Modes: `"empty"` replaces missing vars with `""` (default; correct for MCP arg templates where a missing token is common). `"raise"` raises `KeyError` naming the missing variable. `"leave"` keeps the literal `$FOO` token unchanged. + +**Rationale for `"raise"`**: `root` is a configuration value the operator wrote down. A typo (`$DARNIT_ATT_ROT` instead of `$DARNIT_ATT_ROOT`) should be a loud audit-time error, not a silent expansion to `""` (which would then either error later in a confusing way, or worse, resolve `""` as the current directory). Fail-fast surfaces the misconfiguration where the operator can see it and correlate to the config line. + +**Interaction with tildes**: `os.path.expanduser` (called AFTER `substitute_dollar_vars`) handles `~`. Order matters: substitute vars first, then expand `~`, then resolve to absolute. This preserves the semantics of e.g. `root = "$MY_HOME/darnit"` where `MY_HOME` might itself be `~/.local`. + +**Alternatives considered**: +- `missing="empty"` (matches MCP arg default). Rejected because a silent-empty `root` is a data-integrity risk for attestations. +- `missing="leave"` (keeps `$FOO` literal). Rejected because a literal `$FOO/attestations/` directory landing in someone's tree is worse than a hard error. + +--- + +## R-004: Atomic-write semantics on cross-filesystem `root` + +**Decision**: No change needed. `FilesystemAuditCacheStore` already writes the tempfile into the SAME directory as the target (`stores/defaults/cache.py:53-56`), so a cross-filesystem `root` does not break the rename: + +```python +fd, tmp_path = tempfile.mkstemp( + dir=str(target.parent), # same dir as target -> same filesystem + suffix=".tmp", + prefix="audit-cache-", +) +``` + +`LocalFsAuditCacheStore` inherits this by delegating to `FilesystemAuditCacheStore` with a different `root`. Same-filesystem rename atomicity is preserved even when `root` is on a network mount or a different device than `/tmp`. + +**Rationale**: The pre-feature `darnit.core.audit_cache` module wrote its tempfile into the system tempdir and renamed across filesystems, which failed on Linux with `EXDEV`. Feature 033 already fixed that when it migrated to `FilesystemAuditCacheStore`. This feature inherits the fix; no re-implementation needed. + +**Test coverage**: existing `test_filesystem_defaults.py::test_atomic_rename_leaves_no_tempfile` (feature 033) exercises the tempfile-then-rename path; new `test_local_fs_backend.py` parametrizes it against a `root` outside `/tmp` to double-cover the cross-fs case. + +--- + +## R-005: `_StoreBundle` lazy instantiation with platform-path resolution + +**Decision**: Compatible. Platform-path resolution happens inside the `UserLocal*Store.__init__`, which is called by the factory closure inside `_StoreBundle` on first property access -- not at `resolve_stores()` time. + +**Verified against** `packages/darnit/src/darnit/stores/selection.py`: + +- `resolve_stores()` builds factory closures that capture the backend class and kwargs; it does NOT instantiate. +- `_StoreBundle.project` / `.attestation` / `.report` / `.cache` are `@property` methods that call the factory on first access and memoize. +- Feature-033's `test_us1_lazy_instantiation.py` locks this: a bundle whose `.attestation` is never accessed never constructs the store. + +**Consequence for FR-004 and SC-004**: `UserLocalAttestationStore.__init__` may do `platform.system()`, read `XDG_DATA_HOME` from env, and probe existence -- all safe because it only runs when the audit actually needs the attestation store. Zero-config audits and audits that never touch attestations pay zero cost. + +**Class-shape Protocol validation (SC-007 preservation)**: `resolve_stores()` still does the eager class-shape check via `_protocol_methods()`. Since `UserLocal*Store` inherits or delegates to `LocalFs*Store` (which inherits from `Filesystem*Store`), the class shape validates without any instance construction. + +--- + +## Summary of decisions + +| # | Decision | Impact | +|---|---|---| +| R-001 | Hand-roll platform-path resolvers | ~40 lines, zero new deps | +| R-002 | Reuse `_sanitize_filename` from attestation.py | No copy-paste, SC-005 satisfied by construction | +| R-003 | Use `missing="raise"` for `root` interpolation | Loud-fail on config typos | +| R-004 | Delegate to `FilesystemAuditCacheStore` for atomic write | Cross-fs `root` handled by existing tempfile-in-target-dir | +| R-005 | Platform resolution inside `__init__`, gated by lazy bundle | SC-004 lazy-instantiation invariant preserved | + +None of the five research questions requires further clarification. Ready to consume in Phase 1. diff --git a/specs/034-local-output-store/spec.md b/specs/034-local-output-store/spec.md new file mode 100644 index 00000000..0e37d001 --- /dev/null +++ b/specs/034-local-output-store/spec.md @@ -0,0 +1,207 @@ +# Feature Specification: Local Output Data Store + +**Feature Branch**: `034-local-output-store` + +**Created**: 2026-09-01 + +**Status**: Draft + +**Input**: User description: "I want to develop a local output data store of some kind. Something that can just store stuff like .project, but also output stuff outside the project directory if need be." + +## Clarifications + +### Session 2026-09-01 + +- Q: How does an OSPO leader avoid duplicating `[stores.*]` across every one of their 30 repos? → A: Stay per-repo `.baseline.toml`. Operators template the block via CI/CD, cookiecutter, or a copy-once repo-init tool; env-var interpolation (feature 033's `$VAR` support) makes `root = "$DARNIT_ATT_ROOT"` the escape hatch. No new org- or user-level config file is in scope for this feature. +- Q: Register `local-fs` and `user-local` as one backend with a mode, or as two independently-registered backends? → A: Two independent backends. Each ships its own entry-point registration per artifact kind; `user-local` extends `local-fs` internally to resolve `root` from platform conventions but presents as a distinct `backend = "user-local"` selector in TOML. Class hierarchy stays clean; tests exercise each backend by its published name. +- Q: When an artifact lands outside the repo, does darnit log the resolved absolute path? → A: Yes, at info level, one line per successful outside-repo write, naming the backend, the artifact kind, and the resolved path. Makes CI logs self-documenting and forecloses the "silent misconfiguration" failure mode where an operator can't tell where their attestations went. + +## Context + +Feature 033 (issue #394, PR #396) landed the pluggable per-artifact `Store` Protocols and their filesystem defaults. Every default writes INSIDE the audited repository: + +- `ProjectStateStore` -> `/.project/project.yaml`, `/.project/maintainers.yaml` +- `AttestationStore` -> `/.darnit/attestations/` +- `ReportStore` -> `/.darnit/reports/` +- `AuditCacheStore` -> per-repo hash under system tempdir (already outside) + +That constraint is intentional for `.project/project.yaml` -- it belongs in the repo because it's the maintainer-curated source of truth for project metadata (per the CNCF `.project/` spec). But it is a hard limit for everything else. Today an operator has no way to say: + +- "Write all attestations to `~/.darnit/attestations/` so I can back them up together" +- "Aggregate reports for every repo in this org under `/var/log/darnit//`" +- "Send audit cache to XDG-standard `$XDG_CACHE_HOME/darnit/` instead of hashed system-tempdir paths" +- "Keep sensitive audit output outside the git tree so `git add` can't leak it" + +This spec covers the local outside-repo destination story. It stays within the plugin surface feature 033 defined -- concretely, this becomes a new default backend (or a set of them) that ships in darnit-core alongside the existing in-repo defaults. Remote/network-backed storage (Postgres, S3, Archivista) remains out of scope and stays in issue #391 territory. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - OSPO leader points attestations at a shared local directory (Priority: P1) + +An OSPO leader auditing 30 repos across their org wants every attestation to land in one place they can back up, sync to a paved-road location, or point their SBOM pipeline at. Today each attestation writes into its own repo's `.darnit/attestations/` -- 30 scattered destinations. They want: + +```toml +# Per-repo .baseline.toml (this feature does NOT add an org- or user-level +# config file; see the Clarifications section for the rationale). To avoid +# duplicating this block across 30 repos, operators template it via CI/CD, +# cookiecutter, or a repo-init tool. Env-var interpolation is the escape +# hatch: `root = "$DARNIT_ATT_ROOT"` lets a single per-machine env var +# steer every repo's audits. +[stores.attestation] +backend = "local-fs" +root = "~/darnit-attestations" +``` + +...and have every audit's attestation land there instead of in the repo. + +**Why this priority**: This is the concrete use case that motivated the request. Attestations are the highest-value output to consolidate because they're what an OSPO leader hands to a downstream consumer (regulator, customer, org compliance dashboard). + +**Independent Test**: Configure `[stores.attestation] backend = "local-fs" root = "/tmp/agg"` on a repo, run an audit that emits an attestation, verify the bundle lands at `/tmp/agg/.intoto.json` (or `.sigstore.json`) and NOT in `/.darnit/attestations/`. + +**Acceptance Scenarios**: + +1. **Given** `[stores.attestation] backend = "local-fs" root = "/tmp/attestations"` is set and `/tmp/attestations/` does not yet exist, **When** the audit runs and produces one attestation, **Then** `/tmp/attestations/.` exists and contains the bundle, and the repo's `.darnit/attestations/` is not created. +2. **Given** the same config with an existing non-empty `/tmp/attestations/`, **When** two consecutive audits produce two attestations, **Then** both bundles coexist in the directory and neither overwrites the other. +3. **Given** `root = "~/darnit-attestations"` (tilde-expansion), **When** the audit runs, **Then** the bundle lands at the resolved home path, not literal `~/darnit-attestations`. +4. **Given** the configured `root` is unwritable (permission denied), **When** the audit runs, **Then** the operator sees a clear error naming the backend and the path, and the audit does not silently write to a fallback location. + +--- + +### User Story 2 - Operator redirects reports and cache outside the repo (Priority: P2) + +An operator running darnit on a CI runner (fresh repo checkout per job) wants: + +- The audit-cache to live in a persistent per-runner location so cache hits work across jobs. +- The Markdown/JSON/SARIF reports to land in an artifact directory the runner already knows how to upload. + +Neither destination is inside the repo checkout. + +**Why this priority**: Reports and audit-cache are lower-value to consolidate than attestations (reports are per-run outputs, cache is a performance optimization), but the same abstraction that solves US1 solves both cleanly. Doing them together avoids a second round-trip on the design. + +**Independent Test**: With `[stores.report] backend = "local-fs" root = "$RUNNER_ARTIFACTS/reports"` and `[stores.cache] backend = "local-fs" root = "$RUNNER_CACHE/darnit"`, run an audit twice back-to-back. Verify the Markdown/JSON reports land under the report root and the second run's audit-cache hit is served from the cache root. + +**Acceptance Scenarios**: + +1. **Given** the two configs above, **When** the audit produces a Markdown report, **Then** it lands at `/reports/.md`, not `/.darnit/reports/`. +2. **Given** the same config with `$RUNNER_CACHE/darnit` seeded from a prior run, **When** a fresh audit runs against the same commit, **Then** the cache read hits and the audit skips the sieve loop. +3. **Given** the report `root` contains other files, **When** the audit runs, **Then** it does not delete or reorganize files it did not write. + +--- + +### User Story 3 - Operator selects an XDG-standard location without spelling it out (Priority: P2) + +An operator on Linux wants darnit's default outside-repo behavior to follow the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html): cache in `$XDG_CACHE_HOME/darnit`, data in `$XDG_DATA_HOME/darnit`. On macOS the equivalent is `~/Library/Caches/darnit` and `~/Library/Application Support/darnit`; on Windows it's `%LOCALAPPDATA%\darnit\Cache` and `%LOCALAPPDATA%\darnit\Data`. They want to type one thing: + +```toml +[stores.attestation] +backend = "user-local" +``` + +...and have the backend do the right thing per platform. + +**Why this priority**: Convenience layer on top of US1. Materially reduces the config an operator has to write for the "just put it somewhere sensible outside the repo" case. Not strictly necessary if US1 lands -- the operator can always spell out `$HOME/.local/share/darnit/attestations` explicitly. + +**Independent Test**: On Linux with `XDG_DATA_HOME` unset, configure `[stores.attestation] backend = "user-local"`. Run an audit, verify the bundle lands at `~/.local/share/darnit/attestations/.`. Repeat with `XDG_DATA_HOME=/tmp/xdg` set; verify the bundle now lands at `/tmp/xdg/darnit/attestations/`. + +**Acceptance Scenarios**: + +1. **Given** Linux, `XDG_DATA_HOME` unset, `[stores.attestation] backend = "user-local"`, **When** an audit produces an attestation, **Then** it lands under `~/.local/share/darnit/attestations/`. +2. **Given** Linux, `XDG_CACHE_HOME=/mnt/fast-cache/`, `[stores.cache] backend = "user-local"`, **When** an audit produces a cache write, **Then** the cache file lands under `/mnt/fast-cache/darnit/audit-cache/`. +3. **Given** macOS with the same config, **When** the audit runs, **Then** the destinations follow macOS conventions (`~/Library/Application Support/darnit/`, `~/Library/Caches/darnit/`). + +--- + +### User Story 4 - Filesystem defaults are unchanged unless the operator opts in (Priority: P1) + +An existing darnit user who has never configured `[stores.*]` and never edits their config sees no behavior change. Attestations still land in `/.darnit/attestations/`, reports in `/.darnit/reports/`, cache in the current per-repo tempdir hash. + +**Why this priority**: Backward-compat and constitution I (darnit-core stays predictable). This is a US-shaped invariant, not a feature -- but it needs its own acceptance path so the fix for US1/US2/US3 doesn't accidentally re-home files for the 100% of users who never configured anything. + +**Independent Test**: Run an audit on a repo with no `[stores.*]` config; assert every artifact lands at the exact same path it landed at before this feature. Test both a fresh repo and one with pre-existing `.darnit/attestations/` content. + +**Acceptance Scenarios**: + +1. **Given** no `[stores.*]` block anywhere in the effective config, **When** the audit runs, **Then** attestations, reports, and cache land at the pre-feature default paths. +2. **Given** a `[stores.attestation]` block set to `backend = "local-fs" root = "/tmp/x"`, **When** the audit runs, **Then** only attestations re-home; reports and cache still use the in-repo defaults. + +--- + +### Edge Cases + +- **Tilde (`~`) and `$VAR` in `root`**: users expect these to expand. If they don't, the operator gets a literal directory named `~` in cwd -- surprising and hard to notice until backup time. +- **Symlinks in `root`**: the operator points at `~/attestations` which is a symlink to a network mount. The store should honor the symlink target, not the link. +- **Path escape via `bundle_id`**: a malicious or malformed control emits `bundle_id = "../../etc/passwd"`. The store MUST sanitize so the write stays under `root`. +- **`root` on a different filesystem than the repo**: the store must not assume same-filesystem rename atomicity. `os.rename` across filesystems fails on Linux; write-then-rename in the cache backend needs to be same-filesystem OR degrade gracefully. +- **`root` is a file, not a directory**: distinct error from "does not exist" -- the operator misconfigured. +- **`root` grows unbounded**: no automatic pruning is in scope; the operator manages retention. But the docs should call this out. +- **Multiple concurrent audits sharing the same `root`**: two audit processes running against different repos both write to `~/darnit-attestations/`. Bundle IDs already carry repo/owner so filename collision is unlikely, but the store must not corrupt files on concurrent writes to different filenames. No inter-audit locking is in scope. +- **Windows path handling**: the `root` string may use forward or back slashes; the store should normalize. +- **`.project/` writes**: the CNCF spec says `.project/` belongs in the repo. This feature MUST NOT re-home `.project/` by default, even under `backend = "user-local"`. Explicit `[stores.project]` override remains available but is documented as unusual. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The framework MUST support two new filesystem-backed store families that write outside the audited repository, each independently selectable via `.baseline.toml`'s `[stores.] backend = "..."` selector. The two names are `local-fs` and `user-local`. +- **FR-002**: Both new backends MUST plug into the feature-033 `Store` Protocol surface with no new Protocol methods. +- **FR-003**: The `local-fs` backend MUST accept a `root` config field (absolute path, `~`-expanded home path, or `$VAR`-substituted environment variable) and write every artifact of its kind under that root. +- **FR-004**: The `user-local` backend MUST resolve `root` from platform conventions -- XDG on Linux, Apple support/cache dirs on macOS, LOCALAPPDATA on Windows -- without the operator spelling out a full path. Passing `root` to `user-local` explicitly MUST either be ignored with a warning or rejected with a clear error; the plan phase picks between the two. +- **FR-005**: When an operator configures `[stores.]` to a new outside-repo backend, artifacts of that kind MUST land at the configured location AND MUST NOT also be written to the in-repo default location. +- **FR-006**: When an operator does NOT configure `[stores.]`, behavior for that kind MUST be identical to pre-feature -- same paths, same file names, same on-disk shape. +- **FR-007**: The `root` MUST be created if it does not exist and the operator has permission; missing `root` MUST NOT be an audit failure IF the store's failure semantics per feature 033 are "best-effort" (audit-cache), and MUST surface a clear operator-facing error otherwise. +- **FR-008**: The new backend MUST sanitize identifiers passed as filename components (`bundle_id`, `report_id`, `cache_key`) so that a control cannot cause a write outside `root` via path traversal. +- **FR-009**: The `.project/` project-state store is DIFFERENT: it MUST default to the in-repo `/.project/` even when the operator selects `backend = "user-local"` for other kinds. Overriding `[stores.project]` explicitly remains possible but is documented as unusual. +- **FR-010**: Configuration precedence stays consistent with feature 033: framework-config `[stores.]` block is overridden per-kind by user-config `[stores.]` block; no partial merge within a kind. +- **FR-011**: The framework MUST document each new backend and its config knobs in the plugin-authoring guide (`docs/plugin-authoring/stores.md`) so operators can find them. +- **FR-012**: The framework MUST provide a way for tests and CI to override paths deterministically (e.g., a `DARNIT_STORE_ROOT` env-var interpolation, or per-kind env-var like `DARNIT_ATTESTATION_ROOT`) so parity tests and regression harnesses aren't tied to a real home directory. +- **FR-013**: When a new backend fails to write (permission denied, ENOSPC, etc.), the failure mode MUST match the Protocol's contract per feature 033: attestation/report writes surface as audit-run errors; cache writes are swallowed to a warning log; project-state reads/writes surface as WARN on the affected controls. +- **FR-014**: The new backend MUST NOT introduce any new runtime dependency into darnit-core. +- **FR-015**: After every successful write from `local-fs` or `user-local`, the backend MUST emit an info-level log line naming the backend, the artifact kind, and the resolved absolute path. This applies only to the outside-repo backends; the pre-existing in-repo filesystem defaults are unchanged. + +### Key Entities + +- **`local-fs` backend**: a filesystem-backed backend that writes to any configurable root path, honoring `~` and `$VAR` expansion. Same Protocol surface as the existing `Filesystem*Store` defaults from feature 033; the difference is where `root` gets resolved from. Registered under each `darnit.stores.` entry-point group as `local-fs`. +- **`user-local` backend**: a variant that resolves `root` from platform conventions instead of a config value. Internally extends `local-fs`'s open+write logic and delegates once the platform root is resolved. Registered under each `darnit.stores.` entry-point group as `user-local` so it's a first-class TOML selector, not a hidden mode of `local-fs`. +- **`root` config value**: a string in `.baseline.toml`'s `[stores.]` block. Absolute path, `~`-relative, or `$VAR`-templated. Interpolation MUST already exist per feature 033's shared `darnit.core.env_subst` helper -- confirm at plan time. +- **Path sanitizer**: shared logic that ensures `bundle_id`/`report_id`/`cache_key` cannot cause writes outside `root`. Reuses the sanitizer already in `packages/darnit/src/darnit/stores/defaults/` -- confirm at plan time. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: An operator can point attestations, reports, or cache at any local filesystem location outside the repo with a two-line `.baseline.toml` change (`backend = "..."` + `root = "..."`). No Python, no fork. +- **SC-002**: For every artifact kind configured to a new backend, ZERO writes land at the pre-feature in-repo default path. Verified by an audit-then-grep test that fails if the in-repo path was touched. +- **SC-003**: Zero-config audits (no `[stores.*]` block anywhere) write to the exact same on-disk paths as pre-feature. Verified by the feature-033 US2 zero-config test (`test_us2_zero_config.py`) continuing to pass unchanged. +- **SC-004**: A user-local audit on Linux writes to `$XDG_DATA_HOME/darnit/attestations/` (or `~/.local/share/darnit/attestations/` if `XDG_DATA_HOME` is unset), on macOS to `~/Library/Application Support/darnit/attestations/`, and on Windows to `%LOCALAPPDATA%\darnit\Data\attestations\`. Verified per-platform in CI or with a platform-parameterized unit test. +- **SC-005**: A path-traversal attempt via `bundle_id = "../../etc/foo"` produces a file named exactly `..__..__etc__foo.` (or equivalent sanitized form) under the configured root, and never a write outside root. Verified by fault-injection unit test. +- **SC-006**: The plugin-authoring guide includes a section titled "Writing artifacts outside the repo" with a copy-pasteable `.baseline.toml` snippet for the most-common cases (shared attestation dir, XDG cache, CI runner artifacts dir). +- **SC-007**: `.project/project.yaml` still lands in the audited repo when the operator sets `backend = "user-local"` on OTHER stores but leaves `[stores.project]` unset. Verified by an integration test that asserts `/.project/project.yaml` exists post-audit and no darnit-owned .project file exists under the user-local root. +- **SC-008**: An audit whose configured `root` is unwritable produces a single operator-facing error message that names the backend, the artifact kind, and the resolved path -- and does NOT silently fall back to the in-repo default (would violate feature 033 FR-012, no silent fallback). +- **SC-009**: An audit that writes at least one artifact to `local-fs` or `user-local` produces at least one info-level log line per artifact class, and each log line contains the backend name (`local-fs` or `user-local`), the artifact kind (`attestation`/`report`/`cache`), and the resolved absolute path. Verified by a capsys/caplog assertion on a fixture audit. Zero-config audits produce no such lines. + +## Assumptions + +- The feature-033 `Store` abstraction is the extension point. No new Protocol methods, no framework-level rewiring of the audit driver. +- `.baseline.toml`'s `[stores.]` block already accepts arbitrary backend-specific keys per the `StoreBlock` model (Pydantic `extra = "allow"`). `root` is one such key. No config schema change needed. +- `$VAR` substitution in TOML string values already exists per feature 033's `darnit.core.env_subst` helper. Interpolation applies here without change. +- The audited repo is not the same directory as the user's home. If they are, `user-local` still writes to a resolved home path which happens to be inside the repo -- that's the operator's problem, not a spec issue. +- Cross-filesystem writes (root on a network mount, tempdir on local disk) may lose atomic-rename guarantees on some platforms. This is a filesystem property, not a darnit contract; the store's write should degrade gracefully (e.g., regular write + rename inside the same directory as the target file, not a system tempdir). +- No retention / rotation / TTL logic is in scope. Operators manage their `root` directories themselves. +- `.project/` write-back stays in the repo per the CNCF `.project/` spec's implicit assumption that the file is committable. This is a hard rule, not a default. +- Windows platform coverage is a stretch goal for SC-004; if CI doesn't have a Windows runner, the Windows path resolution is unit-tested with a mocked `os.name` rather than integration-tested. + +## Dependencies + +- Feature 033 (PR #396, merged): the `Store` Protocol surface. This spec has no path forward without it. +- `darnit.core.env_subst` helper (added in 033): used for `$VAR` interpolation in `root` strings. +- Existing per-artifact filesystem defaults in `packages/darnit/src/darnit/stores/defaults/`: the new backends are variants of these, sharing the filename sanitizer and content-type mapping. + +## Out of Scope + +- **A new config layer**. This feature stays on per-repo `.baseline.toml`. No org-level, user-level, or machine-level TOML file, and no `--stores-config` CLI flag. The "one config, many repos" story is a separate feature; operators bridge it today via CI/CD templating, cookiecutter, or env-var interpolation on the existing `.baseline.toml`. +- Remote / network-backed storage (Postgres, S3, Archivista, in-toto-attestation-verifier, GCS, etc.). Those remain issue #391 and are separate backends shipped as plugin packages, not filesystem variants. +- Multi-tenant or org-wide storage patterns beyond "one operator points at one local root." Aggregating across multiple developer machines is out of scope. +- Retention, rotation, or TTL for outside-repo directories. Operators own their storage lifecycle. +- Automatic migration of existing in-repo `/.darnit/` content to a newly-configured outside-repo `root`. If an operator switches configs, old files stay where they were. +- Windows-native path shape testing beyond a unit test with mocked path convention lookup, unless a Windows CI runner is already available at plan time. +- Encryption-at-rest for outside-repo destinations. If sensitive attestation content is a concern the operator uses filesystem-level or LUKS-level encryption; darnit doesn't ship its own. diff --git a/specs/034-local-output-store/tasks.md b/specs/034-local-output-store/tasks.md new file mode 100644 index 00000000..080dd653 --- /dev/null +++ b/specs/034-local-output-store/tasks.md @@ -0,0 +1,184 @@ +--- +description: "Task list for feature 034: local output data store" +--- + +# Tasks: Local Output Data Store + +**Input**: Design documents from `/specs/034-local-output-store/` +**Prerequisites**: [plan.md](plan.md), [spec.md](spec.md), [research.md](research.md), [data-model.md](data-model.md), [contracts/](contracts/), [quickstart.md](quickstart.md) +**Branch**: `034-local-output-store` + +## Format: `[ID] [P?] [Story] Description with file path` + +- **[P]**: Can run in parallel (different files, no dependencies on incomplete tasks) +- **[Story]**: Which user story this task serves (US1, US2, US3, US4). Setup / Foundational / Polish tasks have no story label. +- Every task cites a concrete file path or contract reference so an implementer can act on it in isolation. + +Tests are included per user story because the spec's SC-001..009 are testable acceptance criteria; skipping tests would leave the spec unenforced. + +--- + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: verify the baseline is intact so any regression this feature causes is immediately visible. + +- [X] T001 Verify `git branch --show-current` prints `034-local-output-store` and `git status` is clean. +- [X] T002 Run the feature 033 store test suite to establish the pre-change baseline: `uv run pytest tests/darnit/stores/ -q`. Record the pass count; this feature MUST NOT reduce it. **Baseline: 97 passed.** + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: shared helpers every US depends on. Small; the store classes themselves live in their US phases. + +**⚠️ CRITICAL**: No user story work begins until this phase is complete. + +- [X] T003 Create `packages/darnit/src/darnit/stores/defaults/local_fs.py` with module-level constants and shared helpers ONLY (no concrete `LocalFs*Store` class yet). Specifically: + - `_LOCAL_LOGGER_NAME = "darnit.stores.local"` and a module-level `logger = get_logger(_LOCAL_LOGGER_NAME)`. + - `_resolve_root_config(root: str | Path) -> Path` per R-003 and E-001 root-resolution rules: if `Path`, return as-is; if `str`, call `substitute_dollar_vars(root, missing="raise")`, then `os.path.expanduser`, then `Path(...).resolve()`. Docstring cites data-model.md E-001. + - `_log_wrote(kind_tag: str, backend: str, resolved_path: Path) -> None`: single `logger.info("wrote %s (%s): %s", kind_tag, backend, str(resolved_path))` call. Docstring cites FR-015. + - Import `_sanitize_filename` from `.attestation` for later reuse by test code (do not shadow). +- [X] T004 [P] Create `packages/darnit/src/darnit/stores/defaults/platform_paths.py` skeleton with four public functions (`xdg_data_home`, `xdg_cache_home`, `user_data_root`, `user_cache_root`) that all `raise NotImplementedError` for now. Docstring per data-model.md E-003. Concrete implementation lands in Phase 5. +- [X] T005 [P] Write `tests/darnit/stores/test_local_fs_helpers.py` covering `_resolve_root_config` directly: absolute path pass-through, `~` expansion, `$VAR` interpolation, `$VAR` missing raises `KeyError`, combined `~/$VAR/x`. Uses `monkeypatch.setenv` for env vars. 6-8 test cases. + +**Checkpoint**: local-fs foundation ready. US1 implementation can now begin. + +--- + +## Phase 3: User Story 1 - OSPO leader consolidates attestations (Priority: P1) 🎯 MVP + +**Goal**: `[stores.attestation] backend = "local-fs" root = "/tmp/x"` causes attestations to land at `/tmp/x/.` instead of `/.darnit/attestations/`. This is the OSPO-leader consolidation story and the MVP for this feature. + +**Independent Test**: with only `[stores.attestation] backend = "local-fs" root = "/tmp/agg"` configured, run an audit that emits an attestation. Verify the bundle exists at `/tmp/agg/.` and NOT under `/.darnit/attestations/`. Verify one info log line with `local-fs` and the resolved path. + +- [X] T006 [P] [US1] Add `LocalFsAttestationStore` class to `packages/darnit/src/darnit/stores/defaults/local_fs.py`. Implements `AttestationStore`. `__init__(self, root, **_)` calls `_resolve_root_config` and stores the result; instantiates an internal `FilesystemAttestationStore(root=)` for delegation. `write(bundle_id, bundle_bytes, content_type)` delegates to the internal store, then calls `_log_wrote("attestation", "local-fs", )` on success. `close()` no-op. Reuses `_sanitize_filename` via the delegated store; NO local re-implementation. +- [X] T007 [US1] Add entry-point registration for `local-fs` under `darnit.stores.attestation` in `packages/darnit/pyproject.toml`. Follow the format used by feature 033's `filesystem` registration (search for `[project.entry-points."darnit.stores.attestation"]`). +- [X] T008 [P] [US1] Add `LocalFsAttestationStore` to the re-exports in `packages/darnit/src/darnit/stores/defaults/__init__.py`. Preserve alphabetical order. +- [X] T009 [P] [US1] Write `tests/darnit/stores/test_local_fs_backend.py::TestLocalFsAttestation` with cases: (a) round-trip write + read-back file bytes; (b) content-type -> extension mapping (intoto, sigstore, unknown -> `.bin`); (c) path-traversal sanitization via `bundle_id = "../../etc/foo"` -- assert the resulting on-disk path is `resolved_root / ` AND that `resolved_root.resolve()` is a parent of the actual path (SC-005: verify no directory escape by resolving both sides, not just filename shape); (d) unwritable `root` (e.g. `root = /root/no-perm/` on a non-root test session) raises `StoreOperationError` -- assert the raised exception's message contains the string `"local-fs"`, the string `"attestation"`, AND the resolved absolute path (SC-008: verify the error surface, not just that an error was raised); (e) `root = "$MISSING_VAR"` raises `KeyError` at store construction, before any write. +- [X] T010 [P] [US1] Write `tests/darnit/stores/test_local_fs_logging.py::TestAttestationLogging` with caplog capturing `darnit.stores.local` at INFO. Assert exactly ONE line per write. Assert the message matches `"wrote attestation (local-fs): "`. Assert that a `FilesystemAttestationStore` write emits ZERO lines to this logger (SC-009 zero-config exemption). +- [X] T011 [P] [US1] Write `tests/darnit/stores/test_local_fs_isolation.py::test_only_attestation_redirects`: build a `.baseline.toml` in a `tmp_path` repo with `[stores.attestation] backend = "local-fs" root = ""` and NO other `[stores.*]` blocks. Invoke `resolve_stores`. Verify `bundle.attestation` is a `LocalFsAttestationStore`; `bundle.report`, `bundle.cache`, `bundle.project` are the pre-feature `Filesystem*Store` classes. Do NOT run a full audit (kept scoped to store selection). +- [X] T012 [US1] Add a copy-pasteable snippet to `docs/plugin-authoring/stores.md` under a new "Writing artifacts outside the repo" section, showing the `[stores.attestation] backend = "local-fs" root = "$DARNIT_ATT_ROOT"` example from quickstart § 1. Include the "env-var interpolation is the multi-repo escape hatch" explanation from the spec's Clarifications section. + +**Checkpoint**: US1 complete. `pytest tests/darnit/stores/test_local_fs_backend.py tests/darnit/stores/test_local_fs_logging.py tests/darnit/stores/test_local_fs_isolation.py -q` all pass. `pytest tests/darnit/stores/ -q` overall pass count MUST be >= T002 baseline + these new tests. This satisfies SC-002 (no in-repo writes when redirected), SC-005 (sanitizer), SC-006 (partial: attestation section landed), SC-007 (partial: attestation isolation), SC-008 (unwritable root error), SC-009 (partial: attestation logging). + +--- + +## Phase 4: User Story 2 - CI runner redirects reports and cache (Priority: P2) + +**Goal**: `[stores.report] backend = "local-fs" root = "$RUNNER_ARTIFACTS_DIR/reports"` + `[stores.cache] backend = "local-fs" root = "$RUNNER_CACHE_DIR/darnit"` cause reports and cache to land outside the repo. Second-run cache hits work. + +**Independent Test**: with the two blocks above, run an audit twice back-to-back against the same commit. Verify (a) Markdown/JSON/SARIF reports land under `/reports/`; (b) audit-cache hits on second run because it was written to `/darnit/` and read back from there; (c) neither location is `/.darnit/*`. + +- [X] T013 [P] [US2] Add `LocalFsReportStore` class to `packages/darnit/src/darnit/stores/defaults/local_fs.py`. Same shape as T006 but delegates to `FilesystemReportStore`. `write_markdown` / `write_json` / `write_sarif` each call `_log_wrote(kind_tag, "local-fs", )` with `kind_tag` = `"report:markdown"` / `"report:json"` / `"report:sarif"` respectively. +- [X] T014 [P] [US2] Add `LocalFsAuditCacheStore` class to same file. Delegates to `FilesystemAuditCacheStore`. On successful `write`, log `_log_wrote("cache", "local-fs", )`. `read` does NOT log (it's a no-op on miss and doesn't produce a new file). Failed writes (best-effort per Protocol) do NOT emit the info line; they hit `logger.debug` only, inherited from the delegate. +- [X] T015 [DEFERRED - see plan revision] Originally: `LocalFsProjectStateStore`. Deferred because (a) reimplementing `.project/`-prefix-free YAML I/O is scope creep the plan didn't budget for, (b) the audit driver still uses `repo_path` for non-store work so a redirected project store has weird semantics, and (c) FR-009's canonical answer is "`.project/` stays in-repo". Matches how `user-local` deliberately skips the project registration. Documented as unavailable in T012's docs update rather than shipped-but-unusual. +- [X] T016 [US2] Add TWO entry-point registrations to `packages/darnit/pyproject.toml`: `local-fs` under `darnit.stores.report` and `darnit.stores.cache`. (Not three -- see T015 deferral for why `darnit.stores.project` is skipped.) +- [X] T017 [P] [US2] Extend `packages/darnit/src/darnit/stores/defaults/__init__.py` re-exports to include `LocalFsReportStore`, `LocalFsAuditCacheStore`, `LocalFsProjectStateStore`. +- [X] T018 [P] [US2] Add `TestLocalFsReport` to `tests/darnit/stores/test_local_fs_backend.py`. Cases: (a) round-trip for all three formats; (b) filename extensions correct (`.md`, `.json`, `.sarif`); (c) sanitization of `report_id` (same escape-parent assertion shape as T009 case (c)). About 4 tests. +- [X] T019 [US2] Add `TestLocalFsAuditCache` to the same file (`test_local_fs_backend.py`). NOT parallelizable with T018: both tasks append to the same file, so serialize -- T018 lands first, T019 second. Cases: (a) write then read round-trip; (b) tempfile-then-rename is same-directory (assert on `tempfile.mkstemp` `dir=` argument via monkeypatch OR inspect the file list during a controlled failure); (c) write to unwritable `root` returns None on subsequent read AND does NOT raise (best-effort per Protocol); (d) TTL / staleness passes through to the delegate unchanged (feature 033's existing tests cover this at the delegate level; assert one integration hit here). About 4-5 tests. +- [X] T020 [P] [US2] Add `TestReportLogging` and `TestCacheLogging` to `tests/darnit/stores/test_local_fs_logging.py`. Report logging: three writes -> three log lines with distinct `kind_tag`. Cache logging: successful write -> one line; failed write -> ZERO info lines (but debug line is fine). +- [X] T021 [US2] Extend `tests/darnit/stores/test_local_fs_isolation.py` with `test_report_and_cache_isolation`: `[stores.report]` + `[stores.cache]` set to `local-fs` with different roots, `[stores.attestation]` and `[stores.project]` unset. Assert `bundle.report` is `LocalFsReportStore`, `bundle.cache` is `LocalFsAuditCacheStore`, and the other two are the pre-feature filesystem defaults. +- [X] T022 [US2] Update `docs/plugin-authoring/stores.md` "Writing artifacts outside the repo" section: add the quickstart § 2 CI example. + +**Checkpoint**: US2 complete. Full `pytest tests/darnit/stores/ -q` continues to pass. SC-002, SC-005, SC-006 (report+cache), SC-007 (report+cache isolation), SC-009 (report+cache logging) all satisfied. + +--- + +## Phase 5: User Story 3 - `user-local` with platform-conventional roots (Priority: P2) + +**Goal**: `[stores.] backend = "user-local"` writes to XDG/Apple/LOCALAPPDATA paths without the operator spelling out a `root`. `.project/` remains in-repo (FR-009). + +**Independent Test**: on Linux with `XDG_DATA_HOME` unset, configure `[stores.attestation] backend = "user-local"`. Run an audit that emits an attestation. Verify the bundle lands at `~/.local/share/darnit/attestations/`. Repeat with `XDG_DATA_HOME=/tmp/xdg`; verify it now lands under `/tmp/xdg/darnit/attestations/`. + +- [X] T023 [US3] Implement `xdg_data_home()`, `xdg_cache_home()`, `user_data_root()`, `user_cache_root()` in `packages/darnit/src/darnit/stores/defaults/platform_paths.py`. Follow the platform-dispatch algorithm in data-model.md E-003 and research.md R-001. Handle unknown platforms with XDG fallback. Log the resolved root at debug level for troubleshooting. +- [X] T024 [P] [US3] Write `tests/darnit/stores/test_platform_paths.py` with parametrized coverage: (a) `xdg_data_home()` with `$XDG_DATA_HOME` set + unset; (b) `xdg_cache_home()` same; (c) `user_data_root()` with `platform.system()` monkeypatched to `"Linux"`, `"Darwin"`, `"Windows"`, `"FreeBSD"` (unknown fallback); (d) `user_cache_root()` same set; (e) `LOCALAPPDATA` env var handling for Windows path. About 10 test cases. +- [X] T025 [US3] Create `packages/darnit/src/darnit/stores/defaults/user_local.py` with `UserLocalAttestationStore`, `UserLocalReportStore`, `UserLocalAuditCacheStore`. Each extends its matching `LocalFs*Store`. In `__init__`, ignore any incoming `root` kwarg with a WARNING log line (per data-model.md E-002 and contracts/user-local.md warn-and-ignore section) that includes the resolved platform root. Then call `super().__init__(root=)`. Logging uses backend name `"user-local"`; the `LocalFs*Store` parent's info-log already handles that when the caller passes `backend` -- adjust the info-log helper if needed to accept an override. + - Implementation note: to avoid duplicating log logic, either pass a `backend_name` class attribute (`_BACKEND_NAME = "user-local"` on the subclass) OR override the write methods to call `_log_wrote` with the right backend before delegating. Data-model.md E-005 says shared logger name and format; use the class-attribute approach. +- [X] T026 [US3] Add three entry-point registrations in `packages/darnit/pyproject.toml`: `user-local` under `darnit.stores.attestation`, `darnit.stores.report`, `darnit.stores.cache`. **CRITICAL: do NOT register `user-local` under `darnit.stores.project`** -- this enforces FR-009 at the discovery layer. `resolve_stores` will raise `StoreNotInstalled` if the operator writes `[stores.project] backend = "user-local"`. +- [X] T027 [P] [US3] Add `UserLocalAttestationStore`, `UserLocalReportStore`, `UserLocalAuditCacheStore` to `packages/darnit/src/darnit/stores/defaults/__init__.py` re-exports. +- [X] T028 [P] [US3] Write `tests/darnit/stores/test_user_local_backend.py` with cases: (a) round-trip for each kind on the runtime platform, using an XDG override to keep tests in `tmp_path`; (b) explicit `root` kwarg is warn-and-ignored (assert on the WARNING log via caplog; assert the platform-computed root is used regardless); (c) info-log line uses backend name `"user-local"` (SC-009 backend correctness). About 6-8 test cases. +- [X] T029 [P] [US3] Add `test_user_local_project_state_not_registered` to `tests/darnit/stores/test_user_local_backend.py`: build a `.baseline.toml` with `[stores.project] backend = "user-local"`. Call `resolve_stores`. Assert it raises `StoreNotInstalled` with a message naming `user-local` and `darnit.stores.project`. This locks FR-009 at the discovery layer. +- [X] T030 [P] [US3] Extend `tests/darnit/stores/test_local_fs_isolation.py` with `test_user_local_attestation_project_stays_in_repo`: `[stores.attestation] backend = "user-local"`, no other `[stores.*]` block set. Assert `bundle.attestation` is `UserLocalAttestationStore`, `bundle.project` is `FilesystemProjectStateStore` (in-repo). Confirms SC-007 for the user-local variant. +- [X] T031 [US3] Update `docs/plugin-authoring/stores.md` "Writing artifacts outside the repo" section: add the quickstart § 3 XDG example with the per-platform resolved-path table. + +**Checkpoint**: US3 complete. Full `pytest tests/darnit/stores/ -q` passes. SC-004 (platform paths) fully satisfied via monkeypatched unit tests; a Windows-runner integration pass is a stretch goal (see Polish). + +--- + +## Phase 6: User Story 4 - Zero-config unchanged (Priority: P1) + +**Goal**: an existing darnit user who never configures `[stores.*]` sees byte-for-byte identical behavior. This story has no new production code; it's a verified invariant. + +**Independent Test**: run an audit on a repo with NO `[stores.*]` block. Every artifact lands at the exact same path it landed at before this feature. + +- [X] T032 [US4] Run `uv run pytest tests/darnit/stores/test_us2_zero_config.py -q`. **3/3 pass.** This is the feature-033 witness for the zero-config invariant; it MUST pass unchanged. If it fails, the change to `stores/defaults/__init__.py` or the entry-point registrations broke the invariant -- diagnose before continuing. +- [X] T033 [US4] Write `tests/darnit/stores/test_us4_zero_config_local.py` covering the NEW invariant: with no `[stores.*]` config, none of the new `LocalFs*` or `UserLocal*` entry points are loaded; the `bundle.attestation` / `.report` / `.cache` / `.project` properties still resolve to their pre-feature `Filesystem*Store` classes. Uses `_reset_discovery_cache` to ensure clean state. +- [X] T034 [US4] Add a note to `docs/plugin-authoring/stores.md` "Writing artifacts outside the repo" section: "Zero-config behavior is unchanged. If you do not add `[stores.*]` blocks, artifacts continue to land in `/.darnit/` exactly as before this feature." + +**Checkpoint**: US4 verified. SC-003 satisfied via `test_us2_zero_config.py` continuing to pass. + +--- + +## Phase 7: Polish & Cross-Cutting Concerns + +- [X] T035 [P] Ensure `docs/plugin-authoring/stores.md` "Writing artifacts outside the repo" section is complete: three quickstart examples (T012, T022, T031), the `.project/` FR-009 note, and a "Troubleshooting" subsection matching quickstart § Troubleshooting. Verify against SC-006. +- [X] T036 [P] Run `uv run ruff check .` -- clean. on repo root; MUST exit 0. Auto-fix any lint issues in the files this feature touched; do NOT auto-format unrelated files. +- [X] T037 [P] Run `uv run python scripts/validate_sync.py --verbose` -- all validations pass.; MUST exit 0. This feature introduces no new handlers so the sync check should be a no-op. +- [X] T038 Run the full workspace test sweep -- **3061 passed, 26 skipped, 0 failed**. from repo root: `uv run pytest tests/ -q --deselect tests/darnit/context/test_dot_project_upstream.py::TestUpstreamSpecSync::test_upstream_spec_unchanged`. Confirm exit code 0. Pass count MUST equal the T002 baseline + all new tests added in Phases 3-6 (T005, T009, T010, T011, T018, T019, T020, T021, T024, T028, T029, T030, T033). +- [X] T039 [P] Structure decision guard -- all changes within `stores/defaults/`, `pyproject.toml`, `stores/__init__.py`, `docs/plugin-authoring/stores.md`, `tests/darnit/stores/`.: confirm no file outside `packages/darnit/src/darnit/stores/defaults/`, `packages/darnit/pyproject.toml`, `packages/darnit/src/darnit/stores/__init__.py` (if re-exports move up), `docs/plugin-authoring/stores.md`, and `tests/darnit/stores/` was modified. Command: `git diff --name-only main..HEAD | grep -vE '^(specs/|packages/darnit/src/darnit/stores/defaults/|packages/darnit/pyproject.toml|packages/darnit/src/darnit/stores/__init__.py|docs/plugin-authoring/stores.md|tests/darnit/stores/|CLAUDE.md)'` MUST produce zero lines. +- [X] T040 [P] FR-014 no-new-runtime-dep guard -- pyproject.toml diff is entry-point registrations only, no `[project.dependencies]` change.: `git diff main..HEAD -- packages/*/pyproject.toml` MUST NOT add any entry under `[project.dependencies]`. Entry-point additions under `[project.entry-points.*]` are the only permitted TOML changes. +- [ ] T041 [DEFERRED - no Windows CI runner available] Stretch goal: if a Windows CI runner is available, add an integration test job in `.github/workflows/ci.yml` that runs `test_user_local_backend.py::TestUserLocalAttestation` on Windows and asserts on the `%LOCALAPPDATA%` resolution. Skip this task if no Windows runner exists at implement time; the unit-test coverage via `platform.system()` monkeypatch is sufficient for SC-004. + +**Checkpoint**: feature ready to ship. All 8 success criteria satisfied. Full sweep clean, ruff clean, sync clean, no scope creep outside the planned files. + +--- + +## Dependency graph + +``` +Phase 1 (T001..T002) ── verify baseline + │ + ▼ +Phase 2 (T003..T005) ── shared helpers + skeleton platform_paths + helper tests + │ + ▼ +Phase 3 (T006..T012) ── US1: LocalFsAttestationStore MVP [P1] + │ + ▼ +Phase 4 (T013..T022) ── US2: report + cache + project variants [P2] + │ + ▼ +Phase 5 (T023..T031) ── US3: platform_paths impl + UserLocal* [P2] + │ + ▼ +Phase 6 (T032..T034) ── US4: zero-config invariant verified [P1] + │ + ▼ +Phase 7 (T035..T041) ── polish + guards +``` + +US1..US3 are file-disjoint in their implementation half (T006/T013/T014/T015/T025) so they COULD be authored in parallel. Their tests live in overlapping files (`test_local_fs_backend.py`, `test_local_fs_logging.py`, `test_local_fs_isolation.py`), so serialize on those. + +US4 depends on US1 through US3 all landing (it's a verification story, not new implementation). + +## Parallel opportunities + +Within Phase 3 (US1): +- T006 (class), T008 (re-exports), T009 (backend tests), T010 (logging tests), T011 (isolation test) can be authored in parallel once T003 lands. T007 (pyproject registration) can go in parallel with all of them. + +Within Phase 4 (US2): +- T013, T014, T015 all touch the same `local_fs.py` file -- serialize. +- T017 (init re-exports), T018/T019 (tests in separate files), T020 (logging tests in separate file), T021 (isolation test in separate file), T022 (docs) can go in parallel once the three classes land. + +Within Phase 5 (US3): +- T023 (platform_paths impl) and T024 (platform_paths tests) can go in parallel (T024 was scaffolded in T004). +- T025 (user_local classes) depends on T023. +- T028/T029/T030/T031 can go in parallel once T025 + T026 (registrations) land. + +## Implementation strategy + +**MVP delivery**: Phases 1-3 alone deliver US1 -- OSPO leader can consolidate attestations across many repos. This is publicly usable and worth shipping as an increment if timeline dictates. + +**Incremental after MVP**: Phase 4 (US2 CI operator) is the most common secondary use case; ship next. Phase 5 (US3 XDG defaults) is convenience-layer polish; can ship third or bundle with Phase 4 if timeline allows. Phase 6 (US4 zero-config invariant) always ships in the same increment as the phases that could break it. + +**Skip Phase 7's T041** if no Windows CI runner is available at implement time; note the deferral in the PR body and file a follow-up issue for later Windows validation. diff --git a/tests/darnit/stores/test_local_fs_backend.py b/tests/darnit/stores/test_local_fs_backend.py new file mode 100644 index 00000000..c0b5b244 --- /dev/null +++ b/tests/darnit/stores/test_local_fs_backend.py @@ -0,0 +1,233 @@ +"""Backend tests for `local-fs` outside-repo store variants (feature 034). + +Phase 3 T009 covers `LocalFsAttestationStore`. Phase 4 T018/T019 append +`TestLocalFsReport` and `TestLocalFsAuditCache` to this file. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from darnit.stores.defaults.local_fs import ( + LocalFsAttestationStore, + LocalFsAuditCacheStore, + LocalFsReportStore, +) +from darnit.stores.errors import StoreOperationError + + +class TestLocalFsAttestation: + """T009 cases (a)..(e). Verifies the write path, sanitization, error + surface shape (SC-008), and construction-time failure on missing envvar. + """ + + def test_round_trip_write_and_read_back(self, tmp_path: Path) -> None: + store = LocalFsAttestationStore(root=tmp_path) + payload = b'{"foo": 1}' + store.write("acme-widget", payload, "application/vnd.in-toto+json") + target = tmp_path / "acme-widget.intoto.json" + assert target.exists() + assert target.read_bytes() == payload + + def test_content_type_to_extension_mapping(self, tmp_path: Path) -> None: + store = LocalFsAttestationStore(root=tmp_path) + store.write("a", b"x", "application/vnd.in-toto+json") + store.write("b", b"x", "application/vnd.dev.sigstore.bundle+json") + store.write("c", b"x", "application/octet-stream") # unknown -> .bin + assert (tmp_path / "a.intoto.json").exists() + assert (tmp_path / "b.sigstore.json").exists() + assert (tmp_path / "c.bin").exists() + + def test_path_traversal_sanitization_stays_under_root( + self, tmp_path: Path + ) -> None: + """SC-005: `bundle_id = "../../etc/foo"` MUST NOT escape `root`. + + Assertion is two-part per F2 remediation: the on-disk file is a + direct child of `resolved_root`, and `resolved_root` is a parent + of the actual path when both are resolved. + """ + store = LocalFsAttestationStore(root=tmp_path) + malicious = "../../etc/foo" + store.write(malicious, b"payload", "application/vnd.in-toto+json") + + # Sanitized filename must have no path separators. + expected_name = ".._.._etc_foo.intoto.json" + target = tmp_path / expected_name + assert target.exists(), ( + f"expected file at {target}; got: {list(tmp_path.iterdir())}" + ) + + # Escape-parent check: the actual file lives INSIDE the resolved root. + resolved_root = tmp_path.resolve() + actual = target.resolve() + assert resolved_root in actual.parents or resolved_root == actual.parent, ( + f"path traversal: {actual} escaped {resolved_root}" + ) + + def test_unwritable_root_raises_with_full_error_context( + self, tmp_path: Path + ) -> None: + """SC-008 (per F2 remediation): the raised StoreOperationError's + message MUST contain the backend name, the artifact kind, and + the resolved absolute path. Not just "some OSError happened".""" + # Make root unwritable by creating a read-only parent dir and + # pointing root inside a subdirectory that doesn't exist yet + # (so mkdir(parents=True) triggers PermissionError). + readonly_parent = tmp_path / "readonly" + readonly_parent.mkdir() + readonly_parent.chmod(0o500) # r-x -- no write + + target_root = readonly_parent / "attestations" + store = LocalFsAttestationStore(root=str(target_root)) + + try: + with pytest.raises(StoreOperationError) as exc_info: + store.write( + "acme-widget", + b"{}", + "application/vnd.in-toto+json", + ) + msg = str(exc_info.value) + # SC-008: backend name, artifact kind, and resolved path + assert "local-fs" in msg, f"missing backend in error: {msg!r}" + assert "attestation" in msg, f"missing kind in error: {msg!r}" + # Resolved path -- either the store root or the target file. + # We include the target file in the wrapper, so assert that. + assert str(target_root.resolve()) in msg, ( + f"missing resolved path in error: {msg!r}" + ) + finally: + # Restore perms so pytest can clean up. + readonly_parent.chmod(0o700) + + def test_missing_env_var_raises_at_construction_time( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Case (e): env-var typo raises KeyError at __init__, not lazily + at write time. Fail-fast contract from research R-003.""" + monkeypatch.delenv("DARNIT_TEST_MISSING_ROOT", raising=False) + with pytest.raises(KeyError): + LocalFsAttestationStore(root="$DARNIT_TEST_MISSING_ROOT/x") + + def test_env_var_present_resolves_correctly( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setenv("DARNIT_TEST_ATT_ROOT", str(tmp_path)) + store = LocalFsAttestationStore(root="$DARNIT_TEST_ATT_ROOT/atts") + store.write("id-1", b"payload", "application/vnd.in-toto+json") + assert (tmp_path / "atts" / "id-1.intoto.json").exists() + + def test_close_is_idempotent(self, tmp_path: Path) -> None: + store = LocalFsAttestationStore(root=tmp_path) + store.close() + store.close() + + +class TestLocalFsReport: + """T018: parametric per-format coverage for `LocalFsReportStore`.""" + + def test_round_trip_markdown(self, tmp_path: Path) -> None: + store = LocalFsReportStore(root=tmp_path) + store.write_markdown("run-1", "# Title\n\nBody.") + target = tmp_path / "run-1.md" + assert target.exists() + assert target.read_text() == "# Title\n\nBody." + + def test_round_trip_json(self, tmp_path: Path) -> None: + store = LocalFsReportStore(root=tmp_path) + store.write_json("run-1", '{"summary": {}}') + target = tmp_path / "run-1.json" + assert target.exists() + assert target.read_text() == '{"summary": {}}' + + def test_round_trip_sarif(self, tmp_path: Path) -> None: + store = LocalFsReportStore(root=tmp_path) + store.write_sarif("run-1", '{"$schema":"sarif"}') + target = tmp_path / "run-1.sarif" + assert target.exists() + + def test_report_id_sanitized_no_directory_escape( + self, tmp_path: Path + ) -> None: + """Same escape-parent shape as T009 (c).""" + store = LocalFsReportStore(root=tmp_path) + store.write_json("../../etc/foo", "{}") + # Sanitized: forward slashes -> _, dots kept + expected = tmp_path / ".._.._etc_foo.json" + assert expected.exists() + resolved_root = tmp_path.resolve() + assert resolved_root == expected.resolve().parent + + def test_unwritable_root_raises_with_local_fs_context( + self, tmp_path: Path + ) -> None: + readonly_parent = tmp_path / "readonly-r" + readonly_parent.mkdir() + readonly_parent.chmod(0o500) + target_root = readonly_parent / "reports" + store = LocalFsReportStore(root=str(target_root)) + try: + with pytest.raises(StoreOperationError) as exc_info: + store.write_markdown("r1", "# body") + msg = str(exc_info.value) + assert "local-fs" in msg + assert "report:markdown" in msg + finally: + readonly_parent.chmod(0o700) + + +# T019: NOT parallelizable with T018 (same file). Serialized after T018. +class TestLocalFsAuditCache: + """T019: cache round-trip, best-effort semantics, path safety.""" + + def test_write_then_read_round_trip(self, tmp_path: Path) -> None: + store = LocalFsAuditCacheStore(root=tmp_path) + envelope = {"version": 1, "summary": {"PASS": 5}} + store.write("acme-repo-hash", envelope) + assert store.read("acme-repo-hash") == envelope + + def test_read_miss_returns_none_not_error(self, tmp_path: Path) -> None: + store = LocalFsAuditCacheStore(root=tmp_path) + assert store.read("does-not-exist") is None + + def test_write_to_unwritable_root_swallowed_best_effort( + self, tmp_path: Path + ) -> None: + """Best-effort per Protocol: cache write MUST NOT raise. On + failure, subsequent read returns None.""" + readonly_parent = tmp_path / "readonly-c" + readonly_parent.mkdir() + readonly_parent.chmod(0o500) + target_root = readonly_parent / "cache" + try: + store = LocalFsAuditCacheStore(root=str(target_root)) + # Must not raise. + store.write("k1", {"x": 1}) + # Read must return None (write didn't succeed). + assert store.read("k1") is None + finally: + readonly_parent.chmod(0o700) + + def test_tempfile_lives_in_target_directory( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Cross-fs atomic write safety: `mkstemp` MUST use `dir=`, + not a system tempdir. Verified by capturing the dir arg.""" + import tempfile + + captured = {} + original = tempfile.mkstemp + + def spy_mkstemp(*args, **kwargs): + captured["dir"] = kwargs.get("dir") + return original(*args, **kwargs) + + monkeypatch.setattr(tempfile, "mkstemp", spy_mkstemp) + + store = LocalFsAuditCacheStore(root=tmp_path) + store.write("k", {"x": 1}) + # `dir` argument to mkstemp is `str(target.parent)` == str(tmp_path). + assert captured["dir"] == str(tmp_path.resolve()) diff --git a/tests/darnit/stores/test_local_fs_helpers.py b/tests/darnit/stores/test_local_fs_helpers.py new file mode 100644 index 00000000..8d46495d --- /dev/null +++ b/tests/darnit/stores/test_local_fs_helpers.py @@ -0,0 +1,96 @@ +"""Unit tests for the shared helpers in ``darnit.stores.defaults.local_fs``. + +Feature 034 T005. Covers ``_resolve_root_config`` per data-model E-001 + +research R-003. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from darnit.stores.defaults.local_fs import _resolve_root_config + + +class TestResolveRootConfig: + def test_path_input_returned_verbatim(self, tmp_path: Path) -> None: + """A :class:`Path` argument is a test-only shortcut and is + returned unchanged (no `~` expansion, no `$VAR`).""" + result = _resolve_root_config(tmp_path) + assert result == tmp_path + + def test_absolute_string_resolved_to_path(self, tmp_path: Path) -> None: + result = _resolve_root_config(str(tmp_path)) + assert result == tmp_path.resolve() + + def test_tilde_expansion(self) -> None: + home = Path.home() + result = _resolve_root_config("~/darnit-test-subdir") + assert result == (home / "darnit-test-subdir").resolve() + + def test_env_var_expansion( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setenv("DARNIT_TEST_ROOT", str(tmp_path)) + result = _resolve_root_config("$DARNIT_TEST_ROOT/attestations") + assert result == (tmp_path / "attestations").resolve() + + def test_missing_env_var_raises_keyerror( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + # Ensure the var is unset (in case something else set it). + monkeypatch.delenv("DARNIT_TEST_MISSING", raising=False) + with pytest.raises(KeyError): + _resolve_root_config("$DARNIT_TEST_MISSING/attestations") + + def test_combined_tilde_and_env_var( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Order: substitute $VAR first, then expand ~.""" + monkeypatch.setenv("DARNIT_TEST_SUBDIR", "sub") + home = Path.home() + result = _resolve_root_config("~/$DARNIT_TEST_SUBDIR/leaf") + assert result == (home / "sub" / "leaf").resolve() + + def test_env_var_referencing_home_still_expands( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A `$VAR` whose VALUE contains `~` still gets ~-expanded because + expansion runs AFTER substitution (data-model E-001).""" + monkeypatch.setenv("DARNIT_TEST_HOME_LIKE", "~/darnit") + home = Path.home() + result = _resolve_root_config("$DARNIT_TEST_HOME_LIKE/x") + assert result == (home / "darnit" / "x").resolve() + + def test_bad_type_raises_typeerror(self) -> None: + with pytest.raises(TypeError): + _resolve_root_config(123) # type: ignore[arg-type] + + +class TestPlatformPathsPublicApi: + """Smoke check that the four public helpers are importable and callable. + + Detailed platform-specific behavior lives in + ``test_platform_paths.py`` (T024). + """ + + def test_xdg_data_home_returns_path(self) -> None: + from darnit.stores.defaults.platform_paths import xdg_data_home + + assert isinstance(xdg_data_home(), Path) + + def test_xdg_cache_home_returns_path(self) -> None: + from darnit.stores.defaults.platform_paths import xdg_cache_home + + assert isinstance(xdg_cache_home(), Path) + + def test_user_data_root_returns_path(self) -> None: + from darnit.stores.defaults.platform_paths import user_data_root + + assert isinstance(user_data_root(), Path) + + def test_user_cache_root_returns_path(self) -> None: + from darnit.stores.defaults.platform_paths import user_cache_root + + assert isinstance(user_cache_root(), Path) diff --git a/tests/darnit/stores/test_local_fs_isolation.py b/tests/darnit/stores/test_local_fs_isolation.py new file mode 100644 index 00000000..4f6ecd83 --- /dev/null +++ b/tests/darnit/stores/test_local_fs_isolation.py @@ -0,0 +1,80 @@ +"""Isolation checks: redirecting one kind MUST NOT redirect others. + +SC-007: `.project/` stays in the repo even when other kinds are +routed to outside-repo backends. FR-005: no double-writes. + +Phase 3 T011 covers attestation-only redirect; Phase 4 T021 adds +report+cache; Phase 5 T030 adds the user-local variant. +""" + +from __future__ import annotations + +from pathlib import Path + +from darnit.config.framework_schema import StoreBlock, StoresConfig +from darnit.stores.defaults import ( + FilesystemAttestationStore, + FilesystemAuditCacheStore, + FilesystemProjectStateStore, + FilesystemReportStore, + LocalFsAttestationStore, + LocalFsAuditCacheStore, + LocalFsReportStore, + UserLocalAttestationStore, +) +from darnit.stores.selection import resolve_stores + + +def test_only_attestation_redirects(tmp_path: Path) -> None: + """T011: `[stores.attestation] backend = "local-fs"` and no other + `[stores.*]` block set. Only attestation gets the LocalFs* class; + everything else stays on the pre-feature filesystem default.""" + att_root = tmp_path / "atts-outside" + config = StoresConfig( + attestation=StoreBlock(backend="local-fs", root=str(att_root)), + ) + bundle = resolve_stores(config, repo_path=tmp_path) + + assert isinstance(bundle.attestation, LocalFsAttestationStore) + assert isinstance(bundle.report, FilesystemReportStore) + assert isinstance(bundle.cache, FilesystemAuditCacheStore) + assert isinstance(bundle.project, FilesystemProjectStateStore) + + +def test_report_and_cache_isolation(tmp_path: Path) -> None: + """T021: `[stores.report]` + `[stores.cache]` set to `local-fs`, others + unset. Only those two redirect; attestation + project stay on the + in-repo default (SC-007 for the report+cache pair).""" + report_root = tmp_path / "reports-outside" + cache_root = tmp_path / "cache-outside" + config = StoresConfig( + report=StoreBlock(backend="local-fs", root=str(report_root)), + cache=StoreBlock(backend="local-fs", root=str(cache_root)), + ) + bundle = resolve_stores(config, repo_path=tmp_path) + + assert isinstance(bundle.report, LocalFsReportStore) + assert isinstance(bundle.cache, LocalFsAuditCacheStore) + assert isinstance(bundle.attestation, FilesystemAttestationStore) + assert isinstance(bundle.project, FilesystemProjectStateStore) + + +def test_user_local_attestation_project_stays_in_repo(tmp_path: Path) -> None: + """T030: `[stores.attestation] backend = "user-local"` and no other + `[stores.*]` block set. `bundle.project` is still the in-repo + `FilesystemProjectStateStore` (SC-007 for the user-local variant). + + Uses `resolve_stores` selection only -- we don't run an audit here, + so the platform-computed root of `UserLocalAttestationStore` is + never actually written to.""" + config = StoresConfig( + attestation=StoreBlock(backend="user-local"), + ) + bundle = resolve_stores(config, repo_path=tmp_path) + + assert isinstance(bundle.attestation, UserLocalAttestationStore) + assert isinstance(bundle.project, FilesystemProjectStateStore) + # `bundle.project` uses the audit's `repo_path` as its root, which + # in this test is `tmp_path` -- so `.project/` would live under + # `tmp_path`, NOT under the user-local platform root. + assert bundle.project._repo_path == tmp_path diff --git a/tests/darnit/stores/test_local_fs_logging.py b/tests/darnit/stores/test_local_fs_logging.py new file mode 100644 index 00000000..adf2203f --- /dev/null +++ b/tests/darnit/stores/test_local_fs_logging.py @@ -0,0 +1,126 @@ +"""FR-015 / SC-009 log-line assertions for outside-repo backends. + +Every successful outside-repo write emits exactly one info-level line +to logger `darnit.stores.local` naming the kind, backend, and resolved +path. The in-repo `Filesystem*Store` classes emit ZERO lines to this +logger (zero-config exemption). +""" + +from __future__ import annotations + +import logging +from pathlib import Path + +from darnit.stores.defaults.attestation import FilesystemAttestationStore +from darnit.stores.defaults.cache import FilesystemAuditCacheStore +from darnit.stores.defaults.local_fs import ( + LocalFsAttestationStore, + LocalFsAuditCacheStore, + LocalFsReportStore, +) +from darnit.stores.defaults.report import FilesystemReportStore + +_LOGGER = "darnit.stores.local" + + +class TestAttestationLogging: + def test_one_info_line_per_write( + self, tmp_path: Path, caplog + ) -> None: + store = LocalFsAttestationStore(root=tmp_path) + with caplog.at_level(logging.INFO, logger=_LOGGER): + store.write("acme", b'{"x":1}', "application/vnd.in-toto+json") + records = [r for r in caplog.records if r.name == _LOGGER] + assert len(records) == 1 + assert records[0].levelno == logging.INFO + + def test_message_shape_names_kind_backend_and_path( + self, tmp_path: Path, caplog + ) -> None: + store = LocalFsAttestationStore(root=tmp_path) + with caplog.at_level(logging.INFO, logger=_LOGGER): + store.write("acme", b'{"x":1}', "application/vnd.in-toto+json") + # Format is: "wrote {kind} ({backend}): {path}" + msg = caplog.records[-1].getMessage() + assert msg.startswith("wrote attestation (local-fs): ") + # The resolved path in the message must be inside tmp_path. + expected_target = str((tmp_path / "acme.intoto.json").resolve()) + assert expected_target in msg + + def test_filesystem_default_emits_zero_lines_to_local_logger( + self, tmp_path: Path, caplog + ) -> None: + """SC-009 exemption: the pre-feature `FilesystemAttestationStore` + must NOT emit to `darnit.stores.local`. Zero-config audits stay + log-silent.""" + store = FilesystemAttestationStore(tmp_path) + with caplog.at_level(logging.DEBUG, logger=_LOGGER): + store.write("acme", b'{"x":1}', "application/vnd.in-toto+json") + records = [r for r in caplog.records if r.name == _LOGGER] + assert records == [] + + +class TestReportLogging: + """T020: one info line per format written; three formats = three lines.""" + + def test_each_format_emits_one_line( + self, tmp_path: Path, caplog + ) -> None: + store = LocalFsReportStore(root=tmp_path) + with caplog.at_level(logging.INFO, logger=_LOGGER): + store.write_markdown("run", "# a") + store.write_json("run", "{}") + store.write_sarif("run", "{}") + records = [r for r in caplog.records if r.name == _LOGGER] + tags = [r.getMessage().split(" ")[1] for r in records] + assert tags == ["report:markdown", "report:json", "report:sarif"] + + def test_filesystem_default_report_zero_local_lines( + self, tmp_path: Path, caplog + ) -> None: + store = FilesystemReportStore(tmp_path) + with caplog.at_level(logging.DEBUG, logger=_LOGGER): + store.write_markdown("run", "# a") + assert [r for r in caplog.records if r.name == _LOGGER] == [] + + +class TestCacheLogging: + """T020: cache write emits one info line on success, zero on best- + effort failure (delegate's warning still fires but not to + `darnit.stores.local`).""" + + def test_successful_write_emits_one_line( + self, tmp_path: Path, caplog + ) -> None: + store = LocalFsAuditCacheStore(root=tmp_path) + with caplog.at_level(logging.INFO, logger=_LOGGER): + store.write("k", {"x": 1}) + records = [r for r in caplog.records if r.name == _LOGGER] + assert len(records) == 1 + assert records[0].getMessage().startswith("wrote cache (local-fs): ") + + def test_failed_write_emits_zero_info_lines( + self, tmp_path: Path, caplog + ) -> None: + """Best-effort: swallowed OSError means no target file lands, so + LocalFsAuditCacheStore's post-write `target.exists()` check + skips the info log. Zero lines at INFO level on `darnit.stores.local`.""" + readonly = tmp_path / "readonly" + readonly.mkdir() + readonly.chmod(0o500) + try: + store = LocalFsAuditCacheStore(root=str(readonly / "cache")) + with caplog.at_level(logging.INFO, logger=_LOGGER): + store.write("k", {"x": 1}) + records = [r for r in caplog.records if r.name == _LOGGER] + assert records == [] + finally: + readonly.chmod(0o700) + + def test_filesystem_default_cache_zero_local_lines( + self, tmp_path: Path, caplog + ) -> None: + store = FilesystemAuditCacheStore(tmp_path) + with caplog.at_level(logging.DEBUG, logger=_LOGGER): + store.write("k", {"x": 1}) + assert [r for r in caplog.records if r.name == _LOGGER] == [] diff --git a/tests/darnit/stores/test_platform_paths.py b/tests/darnit/stores/test_platform_paths.py new file mode 100644 index 00000000..bba20624 --- /dev/null +++ b/tests/darnit/stores/test_platform_paths.py @@ -0,0 +1,132 @@ +"""Feature 034 T024: platform-path resolution for the `user-local` backend. + +Per-platform coverage using `monkeypatch.setattr(platform, "system", ...)`. +XDG env-var overrides tested independently. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from darnit.stores.defaults import platform_paths + + +class TestXdgDataHome: + def test_uses_xdg_data_home_when_set( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path)) + assert platform_paths.xdg_data_home() == tmp_path + + def test_falls_back_to_local_share_when_unset( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.delenv("XDG_DATA_HOME", raising=False) + assert platform_paths.xdg_data_home() == Path.home() / ".local" / "share" + + +class TestXdgCacheHome: + def test_uses_xdg_cache_home_when_set( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path)) + assert platform_paths.xdg_cache_home() == tmp_path + + def test_falls_back_to_cache_when_unset( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.delenv("XDG_CACHE_HOME", raising=False) + assert platform_paths.xdg_cache_home() == Path.home() / ".cache" + + +class TestUserDataRoot: + def test_linux_uses_xdg_data( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Linux" + ) + monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path)) + assert platform_paths.user_data_root() == tmp_path / "darnit" + + def test_linux_falls_back_to_home_local_share( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Linux" + ) + monkeypatch.delenv("XDG_DATA_HOME", raising=False) + expected = Path.home() / ".local" / "share" / "darnit" + assert platform_paths.user_data_root() == expected + + def test_macos_uses_apple_application_support( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Darwin" + ) + expected = Path.home() / "Library" / "Application Support" / "darnit" + assert platform_paths.user_data_root() == expected + + def test_windows_uses_localappdata_when_set( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Windows" + ) + monkeypatch.setenv("LOCALAPPDATA", str(tmp_path)) + expected = tmp_path / "darnit" / "Data" + assert platform_paths.user_data_root() == expected + + def test_windows_falls_back_when_localappdata_unset( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Windows" + ) + monkeypatch.delenv("LOCALAPPDATA", raising=False) + expected = ( + Path.home() / "AppData" / "Local" / "darnit" / "Data" + ) + assert platform_paths.user_data_root() == expected + + def test_unknown_platform_uses_xdg_fallback( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "FreeBSD" + ) + monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path)) + assert platform_paths.user_data_root() == tmp_path / "darnit" + + +class TestUserCacheRoot: + def test_linux_uses_xdg_cache( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Linux" + ) + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path)) + assert platform_paths.user_cache_root() == tmp_path / "darnit" + + def test_macos_uses_apple_caches( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Darwin" + ) + expected = Path.home() / "Library" / "Caches" / "darnit" + assert platform_paths.user_cache_root() == expected + + def test_windows_uses_localappdata_cache( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Windows" + ) + monkeypatch.setenv("LOCALAPPDATA", str(tmp_path)) + expected = tmp_path / "darnit" / "Cache" + assert platform_paths.user_cache_root() == expected diff --git a/tests/darnit/stores/test_us4_zero_config_local.py b/tests/darnit/stores/test_us4_zero_config_local.py new file mode 100644 index 00000000..3bb42044 --- /dev/null +++ b/tests/darnit/stores/test_us4_zero_config_local.py @@ -0,0 +1,54 @@ +"""Feature 034 T033 / US4: zero-config invariant preserved. + +With no `[stores.*]` block, `resolve_stores` MUST return the pre- +feature `Filesystem*Store` classes for all four kinds. Neither the +new `LocalFs*Store` nor the new `UserLocal*Store` should be +instantiated. This is the constitutional invariant that lets us ship +outside-repo backends without changing any operator's day-to-day +behavior. +""" + +from __future__ import annotations + +from pathlib import Path + +from darnit.stores import discovery +from darnit.stores.defaults import ( + FilesystemAttestationStore, + FilesystemAuditCacheStore, + FilesystemProjectStateStore, + FilesystemReportStore, + LocalFsAttestationStore, + LocalFsAuditCacheStore, + LocalFsReportStore, + UserLocalAttestationStore, + UserLocalAuditCacheStore, + UserLocalReportStore, +) +from darnit.stores.selection import resolve_stores + + +def test_none_config_yields_filesystem_defaults(tmp_path: Path) -> None: + """None config -> all four kinds resolve to `Filesystem*Store`.""" + discovery._reset_discovery_cache() + bundle = resolve_stores(None, repo_path=tmp_path) + + assert isinstance(bundle.attestation, FilesystemAttestationStore) + assert isinstance(bundle.report, FilesystemReportStore) + assert isinstance(bundle.cache, FilesystemAuditCacheStore) + assert isinstance(bundle.project, FilesystemProjectStateStore) + + +def test_none_config_does_not_instantiate_new_backends(tmp_path: Path) -> None: + """No configured `[stores.*]` -> the new feature 034 backend classes + are never constructed. Confirms FR-006 / SC-003 at the object level.""" + discovery._reset_discovery_cache() + bundle = resolve_stores(None, repo_path=tmp_path) + + for backend in (bundle.attestation, bundle.report, bundle.cache, bundle.project): + assert not isinstance(backend, LocalFsAttestationStore) + assert not isinstance(backend, LocalFsReportStore) + assert not isinstance(backend, LocalFsAuditCacheStore) + assert not isinstance(backend, UserLocalAttestationStore) + assert not isinstance(backend, UserLocalReportStore) + assert not isinstance(backend, UserLocalAuditCacheStore) diff --git a/tests/darnit/stores/test_user_local_backend.py b/tests/darnit/stores/test_user_local_backend.py new file mode 100644 index 00000000..e19d5ee6 --- /dev/null +++ b/tests/darnit/stores/test_user_local_backend.py @@ -0,0 +1,149 @@ +"""Backend tests for `user-local` (feature 034 T028 + T029).""" + +from __future__ import annotations + +import logging +from pathlib import Path + +import pytest + +from darnit.config.framework_schema import StoreBlock, StoresConfig +from darnit.stores.defaults import platform_paths +from darnit.stores.defaults.user_local import ( + UserLocalAttestationStore, + UserLocalAuditCacheStore, + UserLocalReportStore, +) +from darnit.stores.errors import StoreNotInstalled +from darnit.stores.selection import resolve_stores + +_LOGGER = "darnit.stores.local" + + +class TestUserLocalAttestation: + def test_round_trip_at_platform_data_root_for_attestations( + self, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + ) -> None: + """T028 (a): pin the platform data root into `tmp_path` via + `platform.system() -> "Linux"` + `XDG_DATA_HOME=` + so the write lands somewhere reproducible.""" + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Linux" + ) + monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path)) + + store = UserLocalAttestationStore() + store.write("acme", b'{"x":1}', "application/vnd.in-toto+json") + target = tmp_path / "darnit" / "attestations" / "acme.intoto.json" + assert target.exists() + + def test_explicit_root_kwarg_warn_and_ignored( + self, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + caplog, + ) -> None: + """T028 (b): passing `root` to user-local logs a warning and + uses the platform default anyway.""" + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Linux" + ) + monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path)) + bogus = "/never/written/to" + + with caplog.at_level(logging.WARNING, logger=_LOGGER): + store = UserLocalAttestationStore(root=bogus) + + assert any( + "user-local backend ignores" in r.getMessage() + for r in caplog.records + ) + # The store's write path uses the platform root. + store.write("acme", b"x", "application/vnd.in-toto+json") + assert ( + tmp_path / "darnit" / "attestations" / "acme.intoto.json" + ).exists() + assert not Path(bogus).exists() + + def test_info_log_uses_user_local_backend_name( + self, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + caplog, + ) -> None: + """T028 (c): SC-009 backend name correctness. Info log line MUST + include `(user-local)`, not `(local-fs)`.""" + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Linux" + ) + monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path)) + + store = UserLocalAttestationStore() + with caplog.at_level(logging.INFO, logger=_LOGGER): + store.write("acme", b'{"x":1}', "application/vnd.in-toto+json") + + msgs = [r.getMessage() for r in caplog.records if r.name == _LOGGER] + assert any(m.startswith("wrote attestation (user-local): ") for m in msgs) + assert not any("(local-fs)" in m for m in msgs) + + +class TestUserLocalReport: + def test_round_trip_at_platform_data_root_for_reports( + self, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Linux" + ) + monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path)) + + store = UserLocalReportStore() + store.write_markdown("run-1", "# body") + assert ( + tmp_path / "darnit" / "reports" / "run-1.md" + ).exists() + + +class TestUserLocalCache: + def test_round_trip_at_platform_cache_root( + self, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + ) -> None: + monkeypatch.setattr( + platform_paths.platform, "system", lambda: "Linux" + ) + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path)) + + store = UserLocalAuditCacheStore() + store.write("k1", {"x": 1}) + assert ( + tmp_path / "darnit" / "audit-cache" / "k1.json" + ).exists() + assert store.read("k1") == {"x": 1} + + +class TestUserLocalNotRegisteredForProject: + """T029: FR-009 enforced at discovery layer.""" + + def test_stores_project_backend_user_local_raises( + self, tmp_path: Path + ) -> None: + from darnit.stores import discovery + + # Reset cache so a stale earlier lookup doesn't mask the intent. + discovery._reset_discovery_cache() + + config = StoresConfig( + project=StoreBlock(backend="user-local"), + ) + with pytest.raises(StoreNotInstalled) as exc_info: + resolve_stores(config, repo_path=tmp_path) + + err = exc_info.value + # StoreNotInstalled carries `.name` and `.group`. + assert err.name == "user-local" + assert err.group == "darnit.stores.project"