choresdb

choresdb is an experimental macOS automation framework for repeatable, semantic desktop workflows.

The central design rule is:

A demonstration is evidence about an action; it is not the workflow itself.

A recorded click may help identify a Sign in button, but replay must resolve the current button from accessibility, text, visual, and contextual signals. It must not depend on the original screen coordinates.

Current status

This repository contains the first project foundation:

• a typed workflow graph model;
• structural and reachability validation;
• a bounded state-machine runtime;
• a provider-independent perception and target-resolution core;
• a dry-run executor for exercising predefined branches;
• the milestone-one Chrome workflow as data;
• architecture and delivery notes.

Native capture and input primitives are being added incrementally. The first Chrome login slice now uses macOS Accessibility and synthetic keyboard events; screenshot/CV providers are not yet connected.

Quick start

Python 3.11 or newer is required. No runtime dependencies are needed yet.

PYTHONPATH=src python3 -m choresdb validate workflows/milestone_01.json \
  --targets targets/milestone_01.json
PYTHONPATH=src python3 -m choresdb validate-targets targets/milestone_01.json

Exercise the already-authenticated path:

PYTHONPATH=src python3 -m choresdb run workflows/milestone_01.json \
  --dry-run \
  --state detect_auth_state=logged_in

Exercise the login path:

PYTHONPATH=src python3 -m choresdb run workflows/milestone_01.json \
  --dry-run \
  --state detect_auth_state=logged_out \
  --state detect_terms_state=unchecked

Run the tests:

PYTHONPATH=src python3 -m unittest discover -s tests -v

Local command host

The first remote-control milestone includes a foreground, single-session command host. Start it from the project directory:

PYTHONPATH=src python3 -m choresdb host serve

From another terminal, or through SSH into the same Mac and project directory, query it with:

PYTHONPATH=src python3 -m choresdb host status

The host reports protocol version 1 and retains one interactive session across separate CLI invocations. Ask for its executable action catalogue and execute one action with:

PYTHONPATH=src python3 -m choresdb available
PYTHONPATH=src python3 -m choresdb do session.inspect
PYTHONPATH=src python3 -m choresdb do browser.open --url https://example.com
PYTHONPATH=src python3 -m choresdb do auth.check \
  --logged-in-url https://example.com/home \
  --logged-out-url https://example.com/login

For repeatable commands or personal values, generate an executable JSON file, edit it, then pass it back to do:

PYTHONPATH=src python3 -m choresdb template browser.open \
  --output .choresdb/local/browser_open.json
# Edit REPLACE_WITH_HTTP_OR_HTTPS_URL in the generated file.
PYTHONPATH=src python3 -m choresdb do \
  --file .choresdb/local/browser_open.json

PYTHONPATH=src python3 -m choresdb template auth.check \
  --output .choresdb/local/auth_check.json
# Fill or remove the example evidence entries, retaining at least one per state.
PYTHONPATH=src python3 -m choresdb do \
  --file .choresdb/local/auth_check.json

Templates use the versioned shape {"version":1,"action":"...","parameters":{...}} and are derived from the running host's action catalogue. Omitting --output prints the template to standard output. Existing files are protected unless --force is supplied. Generated files use mode 0600; .choresdb/local/ is ignored by Git. Direct parameter flags remain available for quick experiments, but cannot be mixed with do --file.

browser.open currently fixes the browser to Chrome and the opening mode to a new tab. It activates Chrome, creates and navigates a distinct tab, waits for the page to finish loading, then records the window/tab identity as the session's owned browser page. The session moves from idle to page_ready, so another browser.open is not advertised until a later session-reset action exists. session.inspect remains available and reports the current state without exposing the full URL. Build the native helper before executing browser actions:

cd native/ChoresDBMacOS
swift build
cd ../..

macOS may request Accessibility and Chrome Automation permissions on the first run. The URL supplied on the command line can be retained by shell history; daemon state and result metadata retain only its host and a hashed page identity. The catalogue never advertises an action as available before it has an implementation, and each execution updates the session revision and last-result metadata.

After browser.open, auth.check is advertised. It brings the exact owned tab back to the foreground and combines exact URL-state rules with visible accessibility text. Each evidence flag is repeatable:

PYTHONPATH=src python3 -m choresdb do auth.check \
  --logged-in-url https://example.com/home \
  --logged-in-marker "Account settings" \
  --logged-out-url https://example.com/login \
  --logged-out-marker "Continue with Google"

At least one logged-in rule and one logged-out rule are required. A unique match produces logged_in or logged_out; no match or evidence for both sides produces unknown and the session state auth_unknown. Rule values and matched text are not returned—only evidence counts. Closing Chrome or the owned tab ends the interactive session; merely switching tabs is safe because auth.check reselects the owned tab by its stored window and tab IDs.

The host accepts one versioned JSON request at a time through .choresdb/runtime/host.sock. The ignored runtime directory is mode 0700; its socket and process lock are mode 0600. Only one host can own a runtime directory. Use the same explicit --runtime-dir on every command when they run from different working directories. host status, available, and do also support machine-readable --json output.

The first semantic target contracts live in targets/milestone_01.json. Candidate providers may use accessibility, OCR, local CV, or hosted vision, but resolution policy—not a provider—decides whether evidence is strong and unambiguous enough to proceed.

Native Chrome login slice

Build the native helper:

cd native/ChoresDBMacOS
swift build
cd ../..

Copy examples/browser_login.example.json into the ignored local configuration directory, then configure the URL, unique logged-in/logged-out URL or text evidence, federated provider, exact provider account, and explicit terms authorization for the target site. URL state rules match scheme, host, port, and path while ignoring query parameters, fragments, and a trailing slash.

mkdir -p .choresdb/local
cp examples/browser_login.example.json .choresdb/local/browser_login.json
PYTHONPATH=src python3 -m choresdb browser-login \
  .choresdb/local/browser_login.json

Add --debug to follow each node and the accessibility resolution decisions while the workflow runs:

PYTHONPATH=src python3 -m choresdb browser-login \
  .choresdb/local/browser_login.json --debug

Debug output includes state-poll counts, target IDs, confidence, selected element geometry, dispatched actions, and elapsed time. It does not print accessibility text dumps, the configured account identifier, or the full destination URL.

Authenticated text-entry slice

Copy examples/browser_text_entry.example.json into .choresdb/local/ and configure its destination URL, required URL identifier, positive logged-in evidence, and anchor text. The slice aborts unless authentication and URL identity are verified. It then makes at most max_scrolls downward root-page scrolls, selects an unambiguous accessibility text field spatially below the anchor, enters message, verifies the field value, and ends without closing the browser.

Root-page scrolling uses Chrome's bounded AppleScript JavaScript channel so a nested form underneath the pointer cannot consume the scroll. In Chrome, enable View > Developer > Allow JavaScript from Apple Events before running this slice. macOS may also ask whether the terminal or calling application may control Chrome; allow it under System Settings > Privacy & Security > Automation. The native helper exposes only an integer root-scroll operation, not arbitrary workflow-provided JavaScript.

Closing Chrome manually cancels an active browser workflow. Browser-targeted hotkeys, scrolling, and text entry are rejected once Chrome is no longer running, so input cannot fall through to another application.

mkdir -p .choresdb/local
cp examples/browser_text_entry.example.json \
  .choresdb/local/browser_text_entry.json
PYTHONPATH=src python3 -m choresdb browser-text-entry \
  .choresdb/local/browser_text_entry.json --debug

Local personal configuration

Files under .choresdb/local/ are ignored by Git and are the only intended place for personal runtime values such as account identifiers, private URLs, task IDs, and site-specific anchor text. Files under examples/ are sanitized, tracked templates; the CLI refuses to execute an .example.json file directly. Passwords remain forbidden in every configuration.

Before committing, verify a local file is ignored:

git check-ignore -v .choresdb/local/browser_text_entry.json

On the first run, macOS requests Accessibility permission for the native helper. The slice fails closed if both marker sets are visible, neither is visible, a target is ambiguous, checkbox state cannot be verified, the exact configured account is absent, or verification does not reach the logged-in state.

The active slice supports federated sign-in only, initially Google. It can accept an explicitly authorized agreement, activate the declared provider, and select the exact configured account. choresdb never reads, retrieves, stores, or types a password. Chrome, Google Password Manager, iCloud Passwords, or the user owns any password challenge. Signup, account creation, CAPTCHA, and recovery flows are intentionally unsupported.

Intended CLI

The project is working toward these user-facing operations:

choresdb record demo-login-button
choresdb run workflows/milestone_01.json --param url=https://example.com

Passwords are outside the choresdb execution boundary. The browser or password manager handles them; workflows only select a declared provider and account.

See the architecture and roadmap for the current boundaries and implementation sequence.

License

Licensed under the MIT License.