PRD: Bayesian Studio — interactive tuning UI for Bayesian binary sensors

[maxlyth]

Problem Statement

Bayesian binary sensors allow resilient, non-deterministic home automations by combining a broad basket of sensor inputs. When individual sensors fail (configuration errors, connection issues, flat batteries), automations continue to broadly work. However, this resilience makes the sensors hard to tune: finding the right combination of observation weights that produces correct outcomes across real historical behaviour requires a tedious manual loop of editing YAML, triggering a backfill, and visually inspecting probability charts — then repeating. There is no interactive way to explore how weight changes affect historical behaviour, no structured way to annotate what "correct" behaviour looks like, and no overview of which sensors are well-calibrated vs drifting.

Solution

A Home Assistant Supervisor add-on (Bayesian Studio) that provides a visual, interactive environment for tuning Bayesian binary sensors. The add-on connects directly to the HA recorder database and YAML configuration, allowing users to:

• See an overview of all Bayesian sensors with calibration health metrics
• Select any sensor and visualise its probability trace over historical data
• Adjust observation weights interactively and see the simulated trace update in real time
• Annotate ground truth ON/OFF periods and measure prediction accuracy
• Add, remove, and tune observations with an HA-style entity picker
• Save tuned values back to YAML configuration with formatting preserved

User Stories

1. As a HA user, I want to see all my Bayesian sensors listed in one place, so that I can quickly identify which ones may need attention.
2. As a HA user, I want to see calibration health metrics per sensor (always-active observations, never-active observations, observation coverage, fire frequency), so that I can prioritise which sensors to tune first.
3. As a HA user, I want to click a sensor on the overview page and be taken directly to its tuning view, so that I can move from diagnosis to action without navigating menus.
4. As a HA user, I want to see a probability trace chart for any Bayesian sensor over a configurable historical time window, so that I can understand its past behaviour.
5. As a HA user, I want the probability trace to show the threshold line and ON/OFF state bands, so that I can visually correlate probability values with sensor state transitions.
6. As a HA user, I want to adjust prob_given_true and prob_given_false for each observation using sliders, so that I can explore weight changes without editing files.
7. As a HA user, I want the probability trace to update immediately when I move a slider, so that I can see the effect of weight changes against real historical data.
8. As a HA user, I want a simplified "influence" slider as the primary control for each observation, so that I can reason about evidence strength without thinking about two independent probabilities.
9. As a HA user, I want to expand any observation slider to see and edit the raw prob_given_true/prob_given_false values, so that I can make precise adjustments when needed.
10. As a HA user, I want to adjust the sensor's prior and probability_threshold using sliders, so that I can tune the sensor's baseline and decision boundary interactively.
11. As a HA user, I want to annotate time periods as "should be ON" or "should be OFF" in a timeline editor, so that I can define ground truth for accuracy measurement.
12. As a HA user, I want to see accuracy, precision, recall, and F1 metrics computed against my ground truth annotations, so that I can objectively compare different tuning configurations.
13. As a HA user, I want the chart to overlay ground truth periods with colour coding showing correct vs incorrect predictions, so that I can visually identify when the sensor goes wrong.
14. As a HA user, I want to add new observations to a sensor using an entity picker that searches by name, domain, and area, so that I can explore whether additional signals improve accuracy.
15. As a HA user, I want to remove existing observations from a sensor and see the simulated effect before saving, so that I can prune unhelpful observations.
16. As a HA user, I want to save tuned observation weights back to my YAML configuration file with comments and formatting preserved, so that I can apply changes without manual editing.
17. As a HA user, I want the tool to warn me if the YAML file was externally modified since I loaded it, so that I don't accidentally overwrite concurrent changes.
18. As a HA user, I want to reset the tuning page to the original configuration at any time, so that I can discard experimental changes without consequence.
19. As a HA user, I want a "Reload HA config" button after saving, so that I can apply changes to the running system without leaving the tool.
20. As a HA user, I want the add-on to appear in my HA sidebar via ingress, so that I don't need to manage a separate URL or authentication.
21. As a HA user running MariaDB or PostgreSQL as my recorder backend, I want the tool to connect to my database, so that I'm not excluded from using the tool.
22. As a HA user, I want sensor discovery to work regardless of how sensors are defined (YAML, packages, UI config flow), so that I don't need to configure anything manually.
23. As a HA user, I want the overview page to load quickly even with many Bayesian sensors, so that health metrics don't block initial navigation.

Implementation Decisions

Architecture

• New repo: ha-addon-bayesian-studio (this repo). Independent release cycle from the backfill project.
• Engine fork: Pure-Python computation engine forked from homeassistant-bayesian-backfill/core.py, split into focused modules with zero Streamlit imports. Testable locally with pytest (no Docker required).
• UI layer: Streamlit pages import engine modules. Engine never imports Streamlit.
• Distribution: this repo doubles as the HA add-on repository via repository.json. Users add the GitHub URL in Supervisor → Add-on Store.

Engine Modules (bayesian_studio/engine/)

• database.py — SQLAlchemy engine creation; reads recorder DB URL from HA config; read-only connections per dialect (SQLite ?mode=ro, MySQL/PostgreSQL read-only transaction). Pure-Python drivers: pymysql + pg8000.
• state_db.py — load_state_timelines(), get_state_at(), get_attr_at() via SQLAlchemy text() queries
• config_loader.py — get_bayesian_entity_ids(), load_bayesian_config() returning (config_dict, source_info) including file path for YAML sensors, load_location()
• bayes.py — evaluate_observation(), compute_bayesian_probability(), compute_probability_trace() (new: returns trace data, not DB-write rows)
• jinja_env.py — build_jinja2_env(), eval_template()
• solar.py — solar_elevation()

Application Modules (bayesian_studio/)

• ha_client.py — HA REST API wrapper using SUPERVISOR_TOKEN; entity search, area/domain filter, live states, config reload
• influence.py — LLR mapping: influence = log(p_true / p_false); forward mapping fixes p_false, derives p_true; influence slider modifies p_true only
• health.py — calibration health: observation coverage, always/never-active detection, fire frequency
• yaml_writer.py — ruamel.yaml round-trip write-back; optimistic lock via file mtime; backup before write; re-parse to validate
• ground_truth.py — JSON storage per sensor in /addon_config/ground_truth/; accuracy/precision/recall/F1 computation

Sensor Discovery

Via HA REST API (/api/states) filtered to platform: bayesian entities. Fallback: .storage/core.entity_registry direct read. Config loaded from .storage/core.config_entries (UI sensors) or recursive YAML scan (YAML sensors).

Write-Back

• YAML sensors: ruamel.yaml round-trip, source file detected by recursive scan, optimistic lock, backup
• UI sensors: deferred to a later phase (HA REST API config entry options update)
• Template-generated YAML (detected by # GENERATED header or .tmpl sibling): read-only, save button disabled with explanation

Observation Editing

• Add: state and numeric_state platforms only in V1. Template observations deferred.
• Remove: comment-out during pre-release; delete for release. Undo via session state before save.

Performance

• State timelines cached with @st.cache_data(ttl=300) — expensive DB query runs once; Bayes loop recomputes per slider interaction (~500ms estimated for 5000 events × 10 observations)
• Overview health metrics computed lazily in background; basic table loads instantly

Add-on

• config.yaml: ingress: true, ingress_port: 8501, map: [config:rw, addon_config:rw], homeassistant_api: true
• Streamlit served at --server.baseUrlPath matching ingress entry path
• Ingress fallback order if WebSockets fail: (1) HTTP long-polling, (2) direct port exposure

Minimum HA version

2024.1.0 (modern recorder schema: states, states_meta, state_attributes)

Testing Decisions

What makes a good test

Tests verify observable behaviour through public interfaces. They should survive internal refactors — if renaming a private function breaks a test, the test was testing implementation. Tests read like specifications.

TDD approach

New modules are built test-first (RED → GREEN, one behaviour at a time). Ported logic from the backfill project brings existing tests adapted to the new module structure.

Modules with tests

• engine/database.py — DB URL resolution, engine creation, read-only connections per dialect
• engine/bayes.py — Bayes probability computation, observation evaluation (all platforms, all edge cases) — ported from backfill
• engine/state_db.py — timeline loading, forward-fill, 1-day lookback — ported from backfill
• engine/jinja_env.py — Jinja2 mock environment, historical now(), sun.sun elevation — ported from backfill
• engine/solar.py — solar elevation values — ported from backfill
• influence.py — round-trip mapping, negative influence, clamping, zero influence
• yaml_writer.py — round-trip preservation, multi-sensor files, add/remove observations, optimistic lock
• health.py — observation coverage, always/never-active detection, fire frequency
• ground_truth.py — save/load, accuracy/precision/recall/F1, edge cases

Integration tests

Read-only queries against real HA recorder DB; HA REST API client against supervisor endpoint; end-to-end sensor config → compute trace → verify shape.

Out of Scope

• Template (value_template) observation creation in V1 — requires a Jinja2 editor with live preview
• UI (config flow) sensor write-back in V1 — all current sensors are YAML-defined; deferred for broader user support
• Activity/template-generated sensor write-back — these sensors use a separate template generation pipeline
• Multi-user session isolation — single-user tool
• Mobile/responsive layout
• Export/import of ground truth data (Phase 6 polish)

Further Notes

This project is a companion to homeassistant-bayesian-backfill, which retroactively writes Bayesian probability history to the HA recorder. The two tools are complementary: backfill produces the historical data; this studio uses it for interactive tuning. The computation engine is forked rather than shared to allow independent evolution; extraction into a shared library is a future goal once both projects stabilise.