diff --git a/.claude/rules/api-client.md b/.claude/rules/api-client.md new file mode 100644 index 0000000..a211aed --- /dev/null +++ b/.claude/rules/api-client.md @@ -0,0 +1,33 @@ +--- +paths: + - src/lib/api-client.ts + - src/base-command.ts +--- + +# API Client Contract + +## BaseCommand + +`BaseCommand` provides: +- `apiClient` getter — lazy-creates `ApiClient` from config or flag overrides. Errors with exit 1 if no key. +- `requireNumericId(value, name)` — validates string is all digits, errors with exit 2. +- `accountHeaders` getter — returns `{'x-account-id': id}` if `--account-id` set, else `{}`. +- Base flags: `--json`, `--api-key` (hidden), `--api-url` (hidden), `--account-id` (hidden). + +`auth login` extends `Command` directly (not `BaseCommand`) because it works without an existing API key. + +## ApiClient + +- Wraps native `fetch()` — no external HTTP dependencies. +- Methods: `get`, `post`, `patch`, `put`, `delete` — all generic. +- Unwraps V2 envelope: returns `response.data`, not the full `{data, requestId, status}`. +- Auth: `x-api-key` header on every request. Content-Type set only when body is present. +- Errors: parses `errors[]` from response, throws `ApiRequestError(message, statusCode, errors)`. +- Network errors: rethrown as `'Could not connect to API. Check your internet connection.'` + +## Rules + +- Never add external HTTP dependencies (axios, got, node-fetch) — use native `fetch()`. +- Never expose `ApiRequestError` internals (statusCode, raw error array) to the user beyond the message. +- Never bypass `ApiClient` for API calls unless the protocol requires it (SSE streaming). +- The `apiKey` property must not appear in any `this.log()`, `console.log()`, or error output. diff --git a/.claude/rules/commands.md b/.claude/rules/commands.md new file mode 100644 index 0000000..709b8cd --- /dev/null +++ b/.claude/rules/commands.md @@ -0,0 +1,77 @@ +--- +paths: + - src/commands/** + - src/lib/output.ts +--- + +# Command Conventions + +## Structure + +Every command extends `BaseCommand` (exception: `auth login` extends `Command` directly). +Static members are ordered alphabetically: `args`, `description`, `examples`, `flags`. + +```typescript +// Good +export default class DatasetGet extends BaseCommand { + static args = { ... } + static description = 'Get details of a specific dataset' + static examples = [ ... ] + static flags = { ... } + async run(): Promise { ... } +} + +// Bad — wrong order, missing examples +export default class DatasetGet extends BaseCommand { + static description = '...' + static flags = { ... } + static args = { ... } + async run(): Promise { ... } +} +``` + +## Flags and arguments + +- Flag names: kebab-case (`page-size`, `data-source-id`). Body/query params: camelCase (`pageSize`, `dataSourceId`). +- Use `Flags.string()`, `Flags.integer()`, `Flags.boolean()` — match the data type. +- `required: true` on mandatory flags, `options: [...]` for enums, `exclusive: [...]` for mutual exclusion. +- Boolean flags: `default: false`. +- Examples use `<%= config.bin %>` template, never hardcoded `databox`. At least 2 examples per command. +- Args use `Args.string({ required: true })` — even numeric IDs are accepted as strings and validated later. + +## Output by command type + +| Type | Output | Functions | +|---|---|---| +| List | Table + pagination | `formatOutput(data, columns, json)` + `showPagination(pagination, json)` | +| Get / Create / Update | Single record | `formatSingle(data, json)` | +| Delete / Purge / Clear | Confirmation message | `this.log('Resource ID action.')` | +| Set (permissions, timezone) | Confirmation message or single record | `this.log()` or `formatSingle()` | + +## Destructive operations + +Delete, purge, and clear commands require: +1. `--force` flag with `default: false` +2. `confirm()` from `../../lib/prompt.js` when not forced +3. `this.log('Aborted.')` when user declines +4. Success message: `"Resource ID past-tense."` (e.g., `"Dataset 123 deleted."`) + +## Error codes + +- `this.error(msg, {exit: 1})` — general errors (missing auth, API failures) +- `this.error(msg, {exit: 2})` — input validation errors (`requireNumericId`) + +## API calls + +- Always pass `this.accountHeaders` as the last argument to `apiClient.get/post/patch/put/delete`. +- No try/catch around API calls — errors propagate to oclif's handler. +- No direct `fetch()` calls (exception: `ask-genie.ts` for SSE streaming). + +## Update commands + +Update commands with optional flags must guard against empty bodies: +```typescript +if (Object.keys(body).length === 0) { + this.error('Provide at least one field to update (--name or --title).', {exit: 1}) +} +``` diff --git a/.claude/rules/e2e-testing.md b/.claude/rules/e2e-testing.md new file mode 100644 index 0000000..7c6e8be --- /dev/null +++ b/.claude/rules/e2e-testing.md @@ -0,0 +1,105 @@ +# E2E Testing Conventions + +End-to-end suites live in `test/e2e/` and spawn the **built** CLI against a **real** API. +They are separate from the mocked unit suite in `test/commands/`. Full usage: +`test/e2e/README.md`. + +## Boundaries + +| | Unit (`test/commands/*.test.ts`) | E2E (`test/e2e/*.e2e.ts`) | +|---|---|---| +| Runs | `runCommand` in-process | `node bin/run.js` as a child process | +| API | `global.fetch` mocked | Real, over the network | +| Asserts | stdout string contents | exit code + stdout/stderr | +| Command | `npm test` | `npm run test:e2e` | + +The `.e2e.ts` suffix is what keeps them apart — `npm test`'s glob is `test/**/*.test.ts`. +Never rename an e2e file to `.test.ts`, and never add `test/e2e` to `.mocharc.yml`. + +## Rules + +- **Everything goes through the CLI.** No suite makes a direct HTTP call. Setup, + assertions and teardown all use `cli()`. The e2e layer has no API client of its own, + so it cannot drift from the one under test. +- **One file per command group**, named for the group: `test/e2e/.e2e.ts`. +- **Name every created resource** with `e2eName(label)` and register it on a + `ResourceTracker`; tear it down in the suite's `after()`. +- **Always pass `--force`** to destructive commands. Child stdin is `'ignore'`, so an + interactive `confirm()` would hang until the mocha timeout. +- **Match error text with `errorText(result)`**, never `result.stderr` directly — the + CLI hard-wraps messages, so a phrase can be split across lines with padding. +- **Never add mocha `--retries` or `--parallel`.** Mocha retries re-run the whole test, + creating resources twice; parallel suites collide on shared account state. Retry belongs + in `retryRead` (poll a read until the API's cache catches up) and `cliWithRetry` (re-run + a command whose failure matches a transient environment fault). `cliWithRetry` is safe + for assertions too — a genuine failure does not match a transient pattern, and a matched + one is still returned once attempts run out. Shared dev environments do fail in bursts, + including spurious 401s on a valid key. +- **No API key outside `helpers/environments.ts`**, and never a production key. +- **Never mutate a resource the suite did not create without `withRestore()`.** It + records the undo on disk before the change, so an interrupted run can be repaired + with `npm run test:e2e:cleanup`. A bare `finally` does not survive Ctrl-C. + +## Classifying a failure + +A failing e2e test means one of three things. Say which, in the test: + +1. **A CLI defect** — **fix the command.** A skipped test is green and CI cannot tell it + from a passing one, so parking a known-broken command as a skip makes the suite lie. + Only if the fix is genuinely deferred, use `it.skip` with `[BROKEN: ]` in the + title plus a comment giving the API's real contract, the observed symptom, and the + source file that fixes it — and keep that list short. +2. **An environment outage** — `this.skip()` at runtime via `serviceUnavailable(result)`, + which recognises the API's own 5xx/service-down messages. Never hard-code an outage + as expected behaviour. +3. **A capability the account lacks** — `this.skip()` with a logged reason: no agency + account, no Advanced Security add-on, no databoards, no connections. + +A skip must always print or carry its reason. A silent skip is worse than a failure. + +## Verifying a suspected CLI defect + +Before marking anything `[BROKEN]`, confirm it against the raw endpoint with `curl`, +so the report distinguishes a CLI bug from an API one. Two defect families found so far, +both invisible to the unit suite: + +- **Envelope drift** — a command reads `response.items` where the API returns a bare + array, or hands `{items: […]}` straight to `formatOutput`. The unit mock defines the + shape, so it always agrees with itself. Only a real response settles it. +- **Request-body drift** — a command sends a field name the API does not accept + (`interval` vs `syncInterval`, `status` vs `isVerified`, `tags` vs `synonyms`). These + commands could never succeed, and every one of them had a passing unit test. + +**Any command that sends a request body needs both**: an e2e test, and a unit test +asserting the body via `lastBody(method, path)` from `test/helpers.ts`. The unit +assertion is the cheap guard that runs on every `npm test`; the e2e test is what proves +the field names are the ones the API actually wants. + +```typescript +it('sends syncInterval, not interval', async () => { + await runCommand(['dataset', 'set-sync-frequency', '123', '--interval', '60'], {root: process.cwd()}) + expect(lastBody('PUT', '/v2/datasets/123/sync-frequency')).to.deep.equal({syncInterval: 60}) +}) +``` + +Note that `@oclif/test`'s `runCommand` refuses an **empty-string** flag value even though +the real binary accepts one, so assertions about clearing a nullable field belong in the +e2e suite. + +## The API is the source of truth + +`ingestion-api` defines the contract; the CLI follows it. Concretely: + +- A command exposes **every field** of its request contract in + `IngestionApi.Core/Contracts/Request/V2/`, and every query parameter its controller + action declares. +- `--json` returns **what the endpoint returned**. Do not hand-pick a subset into a new + object, and do not reshape values. List commands unwrap `response.items` to a bare + array — that is the one established convention — but the item objects pass through whole. +- Optional string fields are guarded with `!== undefined`, never truthiness, so an empty + string can clear a nullable field. +- Response interfaces mirror the response contract, including inherited members (a + `…Detail` type extends its `…ListItem`). + +Re-derive the mapping from the API source rather than from memory or from the existing +CLI code, and confirm anything surprising against a live endpoint with `curl`. diff --git a/.claude/rules/security.md b/.claude/rules/security.md new file mode 100644 index 0000000..5de73f2 --- /dev/null +++ b/.claude/rules/security.md @@ -0,0 +1,44 @@ +--- +paths: + - src/** +--- + +# Security Rules + +## API key handling + +- `apiClient.apiKey` must never appear in `this.log()`, `console.log()`, or error messages. +- The only legitimate direct access to `apiKey` is in `ask-genie.ts` for the SSE `x-api-key` header. +- Hidden flags (`--api-key`, `--api-url`, `--account-id`) must remain `hidden: true` — they do not appear in help output. + +## Config file + +- Config lives at `~/.config/databox-cli/config.json` with `apiKey` and optional `apiUrl`. +- Never log config file contents. Never weaken file permissions. + +## URL interpolation + +- User-provided IDs are interpolated into API paths: `/v2/datasets/${args.datasetId}`. +- Dataset commands validate with `requireNumericId()` (digits only — safe). +- Other commands do not validate — a malicious ID could produce unexpected API paths. +- New commands that interpolate user input into URL paths should validate the input. + +```typescript +// Safe — validated +this.requireNumericId(args.datasetId, 'Dataset ID') +const response = await this.apiClient.get(`/v2/datasets/${args.datasetId}`, ...) + +// Risk — unvalidated +const response = await this.apiClient.get(`/v2/connections/${args.connectionId}`, ...) +``` + +## Input handling + +- `JSON.parse()` on user-provided flag values (--schema, --records, --data) must be wrapped in try/catch with a user-friendly error message. A bare `JSON.parse` leaks a raw `SyntaxError`. +- `fs.readFileSync` on user-provided paths (--file flag) should check file existence first. +- No unbounded stdin reads without size limits in new commands. + +## Error messages + +- Do not include request headers, full URLs, or API keys in error output. +- `ApiRequestError` messages are displayed to the user — they come from the server and are acceptable. diff --git a/.claude/rules/testing.md b/.claude/rules/testing.md new file mode 100644 index 0000000..275ff43 --- /dev/null +++ b/.claude/rules/testing.md @@ -0,0 +1,54 @@ +--- +paths: + - test/** +--- + +# Testing Conventions + +## Framework + +Mocha + Chai (expect style) + `@oclif/test` (`runCommand`). ESM with ts-node loader. + +## Test structure + +```typescript +import {runCommand} from '@oclif/test' +import {expect} from 'chai' +import {cleanupTestConfig, mockApi, restoreApi, setupTestConfig} from '../../helpers.js' + +describe('domain action', () => { + beforeEach(() => { + setupTestConfig() + mockApi([{ + method: 'GET', + path: '/v2/resources', + response: {status: 'success', requestId: 'test', data: { ... }}, + }]) + }) + + afterEach(() => { restoreApi(); cleanupTestConfig() }) + + it('does the thing', async () => { + const {stdout} = await runCommand(['domain', 'action'], {root: process.cwd()}) + expect(stdout).to.include('expected value') + }) + + it('outputs JSON with --json', async () => { + const {stdout} = await runCommand(['domain', 'action', '--json'], {root: process.cwd()}) + const json = JSON.parse(stdout) + expect(json).to.have.property('expectedKey') + }) +}) +``` + +## Rules + +- **File location mirrors source**: `src/commands/dataset/get.ts` → `test/commands/dataset/get.test.ts` +- **Setup/teardown**: `setupTestConfig()` in `beforeEach`, `cleanupTestConfig()` + `restoreApi()` in `afterEach`. Always both. +- **Mock envelope**: Full `{status: 'success', requestId: 'test', data: {...}}` — not just `{data}`. +- **Realistic mocks**: Include all fields the command accesses, not empty objects. +- **runCommand**: Always pass `{root: process.cwd()}`. +- **Destructive commands**: Use `--force` to skip interactive prompts. +- **Coverage per command**: At minimum one happy-path test + one `--json` test. +- **Error paths**: New validation logic (`requireNumericId`, empty-body guard) needs tests asserting the exit code. +- **Naming**: `describe('domain action')` matches the CLI invocation. `it('verbs behavior')`. diff --git a/.claude/skills/pr-review/SKILL.md b/.claude/skills/pr-review/SKILL.md new file mode 100644 index 0000000..084b40c --- /dev/null +++ b/.claude/skills/pr-review/SKILL.md @@ -0,0 +1,183 @@ +--- +name: pr-review +description: > + Use when asked to review a pull request, branch, or set of changes in this `databox-cli` + repository — "review PR", "review PR #N", "review my changes", "review branch X", a pasted + GitHub PR URL, or "what do you think of this PR". Always use this instead of a single-pass review. +allowed-tools: Agent, Read, Grep, Glob, Bash +--- + +# Comprehensive PR Review + +You are the **review lead**. You coordinate parallel specialist agents, validate their +findings, and synthesize one prioritized review for this TypeScript/oclif CLI project +(~80 commands, each self-contained: BaseCommand → ApiClient → output formatting). + +## Token discipline (non-negotiable) + +**Agents pull; you never push.** Never paste the diff, `.claude/rules/*`, `project-patterns.md`, +or agent instruction files into a prompt. Every agent has Bash/Read — send it the base SHA, +file paths, and the pointers below. You do not need to read the reference files or rules +yourself; you only read `git diff --stat`. + +## Phase 1: Gather context + +### 1. Identify the target and base + +Input may be a PR number, branch, GitHub URL, or "my changes" (current branch vs `master`). + +```bash +# PR number: make sure the head is checked out locally so agents can read source +gh pr checkout # skip if already on the branch +gh pr view --json title,body,baseRefName,additions,deletions --jq '{title,body,baseRefName,additions,deletions}' + +BASE=$(git merge-base HEAD master) # or the PR's baseRefName +git diff $BASE...HEAD --stat +git log $BASE...HEAD --oneline +REVIEW_DIR=/pr-review && mkdir -p $REVIEW_DIR +``` + +Record `BASE` (full SHA) and `REVIEW_DIR` — every agent prompt uses both. + +### 2. Review history (PRs only) + +```bash +gh pr view --json reviews --jq '.reviews[] | {author: .author.login, state, body}' +gh api repos/databox/databox-cli/pulls//comments --jq '.[] | {path, line, body}' +gh api repos/databox/databox-cli/issues//comments --jq '.[] | {author: .user.login, body}' +``` + +If prior reviews exist this is a **re-review (round N)**. Build a **prior-disposition ledger**: +`pattern/location · disposition (fixed | declined | backlogged) · sha/reason/link`. Used in Phase 4. + +### 3. Size tier and rule selection (from `--stat` only) + +| Tier | `src/` lines changed | Team | +|---|---|---| +| **Config-only** | no `.ts` files in `src/` or `test/` | No agents — you review the content directly | +| **Small** | < 100 | Correctness, Testing (+ Consistency if new command files added under `src/commands/`) | +| **Standard** | 100 – 800 | Correctness, Consistency, Testing (+ Security if `lib/`, `base-command.ts`, or `auth/` touched) | +| **Large** | > 800 | All 4 agents; if 15+ command files changed, split Consistency into 2 agents by command domain | + +Rules to name in prompts (by touched path): + +| Touched | Rule file | +|---|---| +| `src/commands/` | `commands.md` | +| `src/base-command.ts`, `src/lib/api-client.ts` | `api-client.md`, `security.md` | +| `src/lib/config.ts`, `src/lib/prompt.ts` | `security.md` | +| `src/lib/output.ts` | `commands.md` | +| `src/commands/auth/` | `security.md` | +| `test/` | `testing.md` | + +All paths are relative to `.claude/rules/`. Every agent gets the same rule list — the +Testing agent always gets `testing.md` in addition. + +## Phase 2: Spawn specialists (one message, parallel) + +`subagent_type: "general-purpose"`. Model: **Correctness, Security, Validator inherit the session model**; +**Consistency, Testing use `model: "sonnet"`** (sufficient for checklist-style verification). +Prompt template — fill the braces, nothing else: + +``` +You are the {AGENT} reviewer for a PR in the databox-cli repo (cwd is the repo root). +Read, in order: +1. .claude/skills/pr-review/references/agents/{agent}.md (your instructions) +2. .claude/skills/pr-review/references/agents/output-format.md +3. .claude/rules/{rule1}.md, .claude/rules/{rule2}.md ... +4. .claude/skills/pr-review/references/project-patterns.md (only your row of the per-agent table matters) +Diff: `git diff {BASE}...HEAD -- {paths}` (all changed files: {file list}) +Investigate beyond the diff (callers, tests, base classes) before reporting. +Write your full report to {REVIEW_DIR}/{agent}.md and return the same text. +PR: "{title}" — {one-line summary of intent}. {Round N re-review | First review}. Tier: {tier}. +``` + +Diff scoping per agent (`{paths}`): + +| Agent | `{paths}` | +|---|---| +| Correctness | `src/` | +| Consistency | `src/` | +| Security | `src/lib/ src/base-command.ts src/commands/auth/` | +| Testing | `test/ src/` | + +Small tier: use `src/ test/` for everyone. + +## Phase 3: Validate + +Skip if no BLOCKER or WARNING was reported. Otherwise spawn **one** validator: + +``` +You are the validator for a PR review. Read .claude/skills/pr-review/references/agents/validator.md +and .claude/skills/pr-review/references/project-patterns.md. +Specialist reports are in {REVIEW_DIR}/*.md — validate only their BLOCKER and WARNING findings. +Diff: `git diff {BASE}...HEAD`. Return only the validation output block. +``` + +Apply verdicts: **CONFIRMED** keeps severity · **DOWNGRADED** drops one level · +**DISMISSED** is removed. + +## Phase 4: Synthesize + +1. **Merge** findings that share a root cause (cite all agents). **Reconcile** against the + ledger: `backlogged`/`declined` → drop silently; `fixed` → keep, annotate + `(regression — previously fixed in )`. +2. **Severity**: BLOCKER (must fix before merge) · WARNING (should fix) · SUGGESTION · PRAISE. +3. **Action per WARNING** — *Fix in PR* only on a positive signal: finding's `In diff: yes`, + or single-file fix under ~20 lines, or regression/test gap introduced by this PR. + Otherwise **Backlog** (pre-existing code, 3+ files, needs a design decision, systemic). +4. **Report** using this shape (omit empty sections): + +```markdown +# PR Review: {title} +_Round N — reconciled against M prior dispositions (K dropped as settled)._ ← re-reviews only + +## Summary +{2-3 sentences: mergeable? strongest / weakest aspect} + +## Verdict: APPROVE | REQUEST CHANGES | COMMENT +**Scope:** B{n} · W{fix}/{backlog} · S{n} · P{n} — {one-line justification} + +## Blockers (N) +### B1: {title} +**File** `path:L42-L55` · **Found by** {agents} · **Confidence** HIGH +**Issue** … **Impact** … **Fix** … (code when useful) + +## Warnings (N) +### W1: {title} +**File** … · **Found by** … · **Confidence** … +**Issue** … **Impact** … **Fix** … +**Action** Fix in PR | Backlog — {matched signal} + +## Suggestions (N) ← one line each: `S1 path:L — issue → fix` +## Praise (N) ← max 3, one line each, specific + +## Coverage +| Agent | Findings | Note | +|---|---|---| + +## Codification candidates (N) ← Phase 6 +``` + +## Phase 5: Triage confirmation + +If warnings exist, close with: + +> Review the **Action** on each warning and reply with overrides or "confirm". I'll then fix +> the *Fix in PR* items (and blockers if you want) and list *Backlog* items for filing. + +Offer, when relevant: fix blockers · write missing tests. + +## Phase 6: Codification candidates + +Run only when there is at least one BLOCKER or WARNING. A candidate is either +**(A) recurred** — 2+ findings share a root pattern across files/agents — or +**(B) known but ungraduated** — matches a `project-patterns.md` entry with no `.claude/rules/` link. +Draft the text in the target file's style (patterns: bold title + context + "Watch for"; +rules: 2–3 lines + Good/Bad example) and ask which to accept. + +## Finding quality bar + +Specific (exact lines) · actionable (concrete fix) · justified (why) · proportional (severity = +impact). No style nits (linting owns formatting), no restating rules, no generic praise, +and frame uncertain intent as a question rather than a finding. diff --git a/.claude/skills/pr-review/references/agents/consistency.md b/.claude/skills/pr-review/references/agents/consistency.md new file mode 100644 index 0000000..1d784d3 --- /dev/null +++ b/.claude/skills/pr-review/references/agents/consistency.md @@ -0,0 +1,67 @@ +# Consistency Agent + +You are the **consistency reviewer** — verify that commands follow the established patterns +and conventions of this oclif CLI project. The codebase has ~80 commands that all follow the +same patterns; drift is the primary risk. Apply the `.claude/rules/` files you were told to +read; don't restate them. + +## What to look for + +**Command structure** +- Class extends `BaseCommand` (exception: `auth login` extends `Command` directly). +- Static members ordered alphabetically: `args`, `description`, `examples`, `flags`. + Every command must have this exact order. +- Class name matches file path: `data-source/get.ts` exports `DataSourceGet`, + `dataset/list.ts` exports `DatasetList`. +- `async run(): Promise` — the only instance method. + +**Flag conventions** +- Flag names: kebab-case (`page-size`, `data-source-id`). Never camelCase. +- Body/query params: camelCase (`pageSize`, `dataSourceId`). Manual conversion in command body. +- `Flags.string()` / `Flags.integer()` / `Flags.boolean()` — correct type for the data. +- `required: true` on mandatory flags, `options: [...]` for enums, `exclusive: [...]` for mutual exclusion. +- Boolean flags use `default: false`. + +**Import conventions** +- All imports use `.js` extension (ESM requirement): `'../../base-command.js'`, not `'../../base-command'`. +- Import ordering: node builtins (`node:fs`) → oclif (`@oclif/core`) → local (`../../base-command.js`). +- Only import what's used. Destructured imports from oclif: `{Args, Flags}`, `{Args}`, `{Flags}` — only what's needed. + +**Examples** +- Use `<%= config.bin %>` template syntax, never hardcoded `databox`. +- At least 2 examples per command (basic usage + `--json` or variant). +- Examples demonstrate realistic usage, not just flag enumeration. + +**Output formatting by command type** +- List: `formatOutput(data, columns, json)` + `showPagination(pagination, json)`. +- Get / Create / Update: `formatSingle(data, json)`. +- Delete / Purge / Clear: `this.log('Resource ID action.')` — no `formatSingle`. +- Set operations: `this.log()` confirmation or `formatSingle()`. + +**Destructive operations** +- `--force` flag with `default: false`. +- `confirm()` from `../../lib/prompt.js` when not forced. +- `this.log('Aborted.')` when user declines. +- Success: `"Resource ID past-tense."` (e.g., `"Dataset 123 deleted."`, `"Data source 456 purged."`). + +**Error codes** +- `this.error(msg, {exit: 1})` — general errors. +- `this.error(msg, {exit: 2})` — input validation errors. + +**API client usage** +- Always pass `this.accountHeaders` as the last argument to API calls. +- No try/catch around API calls (except `auth login`). +- No direct `fetch()` calls (except `ask-genie.ts`). + +**Description text** +- `static description` is a short sentence fragment (no period, starts with verb or noun). +- Flag `description` properties are short, start lowercase after the flag name. + +## How to review + +1. Compare the changed command's structure against 2-3 existing commands of the same type. +2. Check static member ordering is alphabetical. +3. Verify flag naming (kebab-case) vs body/query property naming (camelCase). +4. Confirm the right output function is used for the command type. +5. Verify examples use `<%= config.bin %>` and are realistic. +6. For new commands, check they follow the canonical pattern for their type (list/get/create/update/delete/set). diff --git a/.claude/skills/pr-review/references/agents/correctness.md b/.claude/skills/pr-review/references/agents/correctness.md new file mode 100644 index 0000000..103198d --- /dev/null +++ b/.claude/skills/pr-review/references/agents/correctness.md @@ -0,0 +1,53 @@ +# Correctness Agent + +You are the **correctness reviewer** — find bugs, logic errors, edge cases, and runtime +behaviour that would misbehave in production or give users confusing errors. +Apply the `.claude/rules/` files you were told to read; don't restate them. + +## What to look for + +**JSON.parse on user input without try/catch** +- `JSON.parse(flags.xxx)` on user-provided values (`--schema`, `--records`, `--data`, + `--date`, `--measure`, `--tags`, `--columns`) must be wrapped in try/catch with + `this.error('Invalid JSON for --flag: ...', {exit: 2})`. +- A bare `JSON.parse` leaks a raw `SyntaxError` with no actionable message. +- `ask-genie.ts` wraps its `JSON.parse` correctly — that is the model to follow. + +**Missing requireNumericId validation** +- Dataset commands call `requireNumericId()`. Other commands interpolate args straight into + URL paths (`/v2/connections/${args.connectionId}`) without validation. +- New commands taking resource IDs as args should validate or document why the ID may be non-numeric. + +**Missing empty-body guard in update commands** +- Update commands with optional flags must check `Object.keys(body).length === 0` and error + with exit code 1. Some existing updates have this, some don't — new ones must. + +**Error propagation** +- Commands intentionally do NOT wrap API calls in try/catch — errors propagate to oclif's handler. +- Verify new commands maintain this pattern: no swallowed errors, no redundant catch blocks. +- Exception: `auth login` has a try/catch for validation (intentional — validation failure is non-fatal). + +**Edge cases** +- Empty string args (e.g., `databox dataset get ""`) — accepted by oclif, passed to API as empty path segment. +- Negative page numbers — `Flags.integer()` accepts negatives with no guard. +- `fs.readFileSync` in `dataset ingest --file` with no file-existence check — raw Node error. +- Stdin detection via `!process.stdin.isTTY` — may incorrectly detect piped input in some environments. + +**Double-parse inconsistency** +- `BaseCommand.init()` parses flags into `this.flags`. Many commands also `await this.parse(ClassName)` + in `run()` to destructure `{args, flags}` locally. +- Mixing `flags.xxx` (local) and `this.flags.xxx` (from init) for the same data is fragile. + Watch for new commands that reference both for overlapping flag names. + +**Type safety** +- Interfaces defined inline per command — verify the interface matches what the API actually returns. +- Optional/nullable fields should use `| null` or `?`, not assume presence. +- `Flags.integer()` returns `number | undefined` — check for `undefined` before using in arithmetic. + +## How to review + +1. Read surrounding code at each changed location, not just the hunk. +2. For new commands, trace the full flow: flag parsing → validation → API call → output formatting. +3. Check if the command type (list/get/create/update/delete/set) follows its established sub-pattern. +4. Look for `JSON.parse` without try/catch on any user-provided input. +5. Check that every user-provided ID interpolated into a URL path is validated. diff --git a/.claude/skills/pr-review/references/agents/output-format.md b/.claude/skills/pr-review/references/agents/output-format.md new file mode 100644 index 0000000..bed774d --- /dev/null +++ b/.claude/skills/pr-review/references/agents/output-format.md @@ -0,0 +1,34 @@ +# Common Output Format + +Return **only** this block — no preamble, no restated rules, no quoted diff. + +``` +## [Agent Name] Review + +### Summary +[1-2 sentences: what you examined, overall assessment] + +### Findings + +#### [BLOCKER|WARNING|SUGGESTION|PRAISE] - [Short title] +- **File**: `src/commands/dataset/list.ts:L42` (or `L42-L55`) +- **In diff**: yes | no (is the flagged line inside a hunk this PR changes?) +- **Confidence**: HIGH | MEDIUM | LOW +- **Description**: [what the issue is] +- **Why it matters**: [impact — what goes wrong, who is affected] +- **Suggested fix**: [concrete recommendation; code only if it clarifies] + +### No-Issue Confirmation +[≤ 5 one-line bullets: areas checked that were clean] +``` + +## Budget + +- At most **8 findings**. If you have more, keep the highest severity / confidence and fold + the rest into one SUGGESTION ("also: …"). +- At most **2 PRAISE**, and only for something specific and non-obvious. +- Merge findings that share a root cause into one entry listing all locations. +- Only report what you verified by reading surrounding code — LOW confidence means you could + not verify, not that you didn't look. +- Do **not** run `npm test` / `npm run build` — CI and the review lead handle that. Spend + your budget reading code. diff --git a/.claude/skills/pr-review/references/agents/security.md b/.claude/skills/pr-review/references/agents/security.md new file mode 100644 index 0000000..c27807b --- /dev/null +++ b/.claude/skills/pr-review/references/agents/security.md @@ -0,0 +1,60 @@ +# Security Agent + +You are the **security reviewer** — find API key exposure, input sanitization issues, +config file security risks, and unsafe URL interpolation. Apply the `.claude/rules/` files +you were told to read; don't restate them. + +## What to look for + +**API key exposure** +- `apiClient.apiKey` must never appear in `this.log()`, `console.log()`, or error messages. +- The only legitimate access to `apiClient.apiKey` is in `ask-genie.ts` for the SSE header. +- Verify no new commands access `apiClient.apiKey` directly. +- Search for string literals containing "apiKey", "api-key", "x-api-key" in log/error output. + +**Hidden flags** +- `--api-key`, `--api-url`, `--account-id` must remain `hidden: true` in `BaseCommand.baseFlags`. +- Verify no command overrides these flags without `hidden: true`. + +**Config file security** +- Config at `~/.config/databox-cli/config.json` contains the API key. +- Never log config file contents or path with key. +- No changes that expose the config file to stdout or error output. + +**URL path interpolation** +- User-provided IDs interpolated into paths: `/v2/datasets/${args.datasetId}`. +- With `requireNumericId` (digits only) this is safe. +- Without validation, a user could pass values containing `/`, `?`, or `..` — producing unexpected API paths. +- New commands that interpolate user input into URL paths must validate the input. + +```typescript +// Safe +this.requireNumericId(args.datasetId, 'Dataset ID') + +// Risk — connectionId could contain path-traversal characters +await this.apiClient.get(`/v2/connections/${args.connectionId}`, ...) +``` + +**Bypassing ApiClient** +- Any `fetch()` call not going through `ApiClient` must be justified (currently only `ask-genie.ts` for SSE). +- New direct `fetch()` calls must not hard-code API keys, must handle errors, must not log headers. + +**Input sanitization** +- `JSON.parse` on user flags without try/catch leaks raw `SyntaxError` stack traces. +- `fs.readFileSync` on user-provided paths — verify no path traversal concern for the use case. +- Stdin reads — verify no unbounded memory allocation. + +**Error message content** +- Error messages displayed to the user must not contain: + - Request headers (especially `x-api-key`) + - Full URL paths with embedded credentials + - Stack traces (except via oclif's default error handler in debug mode) +- `ApiRequestError` messages from the server are acceptable — they're designed for end users. + +## How to review + +1. Search changed files for `apiKey`, `api-key`, `config`, `fetch(`, `process.env`. +2. Check that hidden flags remain hidden. +3. Verify any user input going into URL paths is validated. +4. Look for new `fetch()` calls bypassing `ApiClient`. +5. Check error messages for internal details. diff --git a/.claude/skills/pr-review/references/agents/testing.md b/.claude/skills/pr-review/references/agents/testing.md new file mode 100644 index 0000000..59c2ce9 --- /dev/null +++ b/.claude/skills/pr-review/references/agents/testing.md @@ -0,0 +1,73 @@ +# Testing Agent + +You are the **testing reviewer** — find coverage gaps, mock quality issues, and test +structure problems. Apply the `.claude/rules/` files you were told to read; don't restate them. + +## What to look for + +**Coverage gaps** +- Every new command in `src/commands/` must have a corresponding test at + `test/commands//.test.ts`. +- Every new command must have at least one happy-path test and one `--json` test. +- New validation logic (`requireNumericId`, empty-body guard, `JSON.parse` wrap) needs + tests asserting the correct exit code. +- New flags should have at least one test exercising them. +- Error paths: if the command has a `this.error()` call, there should be a test that + triggers it. + +**Mock quality** +- Mock responses must include the full V2 envelope: + `{status: 'success', requestId: 'test', data: {...}}`. + Missing `status` or `requestId` will not break tests today but violates the contract. +- The `data` field must match the response type the command expects — not just `{}`. + Include all fields the command accesses in its `formatOutput`/`formatSingle` call. +- Mock HTTP method must match what the command actually calls (`GET`, `POST`, `PATCH`, `PUT`, `DELETE`). +- Mock path must match the exact API path including any interpolated IDs. + +```typescript +// Good — realistic mock +mockApi([{ + method: 'GET', + path: '/v2/datasets/123', + response: {status: 'success', requestId: 'test', data: { + id: 123, title: 'Revenue', dataSourceId: 456, createdAt: '2024-01-01', + timezone: 'UTC', primaryKey: null, schema: null, + }}, +}]) + +// Bad — empty data, missing fields +mockApi([{ + method: 'GET', + path: '/v2/datasets/123', + response: {status: 'success', requestId: 'test', data: {}}, +}]) +``` + +**Test structure** +- `setupTestConfig()` in `beforeEach`, `cleanupTestConfig()` + `restoreApi()` in `afterEach`. + Both cleanup calls are required — missing either leaks state. +- `runCommand()` always includes `{root: process.cwd()}`. +- `describe('domain action')` naming matches CLI invocation (e.g., `'dataset list'`). +- `it('verbs behavior')` naming (e.g., `'lists datasets'`, `'deletes with --force'`). + +**Destructive command tests** +- Delete/purge/clear tests must use `--force` to skip interactive prompts. +- Verify the success message matches the pattern: `"Resource ID action."`. + +**Missing test cases to flag** +- List commands: test with empty results (`items: []`). +- Commands with pagination: test that `showPagination` output appears. +- Commands with optional flags: test the default behavior (no flags) and with flags. + +**Tests impacted by diff** +- If a command's interface or output format changed, check that its test still validates + the new shape. +- If `test/helpers.ts` changed, check that all tests still work with the new helpers. + +## How to review + +1. For each changed command file, verify a corresponding test file exists and was updated. +2. Check mock responses include the full API envelope and realistic data. +3. Verify destructive commands test with `--force`. +4. Look for missing error path tests. +5. Check that test names match conventions (`describe`/`it` naming). diff --git a/.claude/skills/pr-review/references/agents/validator.md b/.claude/skills/pr-review/references/agents/validator.md new file mode 100644 index 0000000..eda8cb6 --- /dev/null +++ b/.claude/skills/pr-review/references/agents/validator.md @@ -0,0 +1,51 @@ +# Validator Agent + +You are the **adversarial validator** — the last line of defense against false positives. +A finding is guilty until proven innocent: try to disprove each one against the real code +and confirm only where you cannot find a reasonable counter-argument. + +## Inputs + +- Specialist reports in the review directory you were given (`*.md`). Validate **only** + BLOCKER and WARNING findings; ignore SUGGESTION and PRAISE. +- The diff command you were given. Run it once, scoped to the files the findings cite. +- `project-patterns.md` — a finding that matches a documented pattern is evidence toward CONFIRMED. + +## For each BLOCKER / WARNING + +1. Open the cited file at the cited lines, plus ~50 lines of context each side. +2. Look for mitigation the specialist may have missed: + - callers guarding the input; oclif flag validation (`required`, `options`) handling it + - base class (`BaseCommand`) providing defaults or validation + - the same pattern used safely elsewhere in the codebase + - for "missing test" findings: grep the test directory for the command name and claimed + behaviour before confirming — the test may live in a sibling file +3. Judge: is it real? is the severity right? would the suggested fix break something? + +Order: BLOCKERs first, then WARNINGs; spend the most effort on LOW/MEDIUM confidence +findings and on findings reported by multiple agents (same root cause?). + +## Verdicts + +- **CONFIRMED** — real, no mitigation found, severity appropriate. Add evidence if you found more. +- **DOWNGRADED** — merit, but partial mitigation or rare path; state the lower severity. +- **DISMISSED** — handled elsewhere. Cite the file:line that disproves it. + +## Output + +``` +## Validation Results + +### Finding: [original title] +**Original severity**: BLOCKER|WARNING · **Reported by**: [agents] +**Verdict**: CONFIRMED|DOWNGRADED|DISMISSED +**Evidence**: [file:line references] +**Reasoning**: [specific, not generic] + +### Validation Summary +Validated N · Confirmed N · Downgraded N · Dismissed N — [one sentence on signal quality] +``` + +Don't rubber-stamp ("looks correct" is useless), don't dismiss because a pattern is common, +and don't add new findings except under a brief "Additional concerns" if something critical +was clearly missed. diff --git a/.claude/skills/pr-review/references/project-patterns.md b/.claude/skills/pr-review/references/project-patterns.md new file mode 100644 index 0000000..6f25516 --- /dev/null +++ b/.claude/skills/pr-review/references/project-patterns.md @@ -0,0 +1,115 @@ +# Project-Specific Patterns and Pitfalls + +Known recurring issues and patterns specific to this codebase. Review agents should check for +these actively — they represent real bugs and review feedback, not hypothetical concerns. + +--- + +## Critical + +**Fields guessed from the endpoint name instead of read from the C# contract** +The single largest source of real bugs in this repo. Six commands sent request bodies the API +silently ignored (`interval` for `syncInterval`, `status` for `isVerified`, `tags` for +`synonyms`), four crashed rendering a response shape that never existed, and six table columns +were permanently blank. **Every one of them had a passing unit test**, because the mock encoded +the same guess as the code, so the test and the bug agreed with each other. +- **Watch for**: any request body field, response interface member or table column that cannot + be traced to `ingestion-api/src/IngestionApi.Core/Contracts/{Request,Response}/V2/`. Check the + contract, not the endpoint name, and not the existing CLI code. +- **Watch for**: envelope assumptions — `response.items` where the endpoint returns a bare + array, or `{items: […]}` handed straight to `formatOutput`. +- A command that sends a body needs **both** an e2e test and a unit test asserting the body via + `lastBody(method, path)`. A unit test alone cannot catch this class of bug. +- Inheritance counts: a `…Detail` response extends its `…ListItem`, so a detail view that + returns fewer fields than a list row is a defect, not a design choice. + +**Unwrapped JSON.parse on user-provided flag values** +~10 commands parse flag values with bare `JSON.parse()` (`--schema`, `--records`, `--data`, +`--date`, `--measure`, `--tags`, `--columns`). Malformed JSON produces a raw `SyntaxError` +with no actionable message. Only `ask-genie.ts` wraps its `JSON.parse` correctly. +- **Watch for**: any `JSON.parse(flags.xxx)` or `JSON.parse(args.xxx)` without a try/catch + that calls `this.error('Invalid JSON for --flagname: ...', {exit: 2})`. + +```typescript +// Bad — raw SyntaxError to user +body.schema = JSON.parse(flags.schema) as SchemaType + +// Good — user-friendly error +try { + body.schema = JSON.parse(flags.schema) as SchemaType +} catch { + this.error('Invalid JSON for --schema. Expected format: [{"columnId":"...","dataType":"..."}]', {exit: 2}) +} +``` + +--- + +## High + +**A flag rename that does not sweep every surface** +Renaming a flag touches five places, and a PR that updates only the first is worse than one that +renames nothing — the docs then actively mislead. `--tags`→`--synonyms` and `--key`→ +`--integration-key` each left stale references behind, and a review was filed against the +*correct* README on the assumption a rename had happened that had not. +- **Watch for**: a changed flag name in `src/commands/` with no matching change in `test/`, + `README.md` (regenerate with `npx oclif readme`), `skills/databox-*/SKILL.md` and + `CHANGELOG.md`. Grep the old name across the repo; the count should be zero. +- Before reporting a rename as a bug, confirm the old flag is actually gone. Different commands + legitimately use different names for the same concept (`--data-source-id` on `dataset create`, + `dataset list` and `metric data`; `--source-id` on `metric list`). + +**Double-parse inconsistency** +`BaseCommand.init()` parses flags into `this.flags`. Most commands with args also call +`this.parse(ClassName)` in `run()` to destructure `{args, flags}` locally. Some commands +reference `flags` (local) for domain flags but `this.flags` (from init) for base flags +like `json` and `account-id`. This works today but is fragile. +- **Watch for**: mixing `flags.xxx` and `this.flags.xxx` in the same command for the same + or overlapping data. + +--- + +## Medium + +**Empty update bodies** +Update commands whose flags are all optional must refuse to send an empty PATCH and name the +flags they wanted. Every such command now does; `test/validation/empty-body.test.ts` sweeps them +and its last test walks `src/commands` to assert the table still covers every guard. +- **Watch for**: a new all-optional update command that builds a body conditionally and never + checks whether it stayed empty, or one added without a row in that sweep. +- The guard is **exit 1**, not exit 2, per `.claude/rules/commands.md` — the command is + well-formed, it just has nothing to do. Do not report this as an exit-code bug. + +--- + +## Low + +**ask-genie bypasses ApiClient** +`analyze/ask-genie.ts` makes a direct `fetch()` call to a different service URL, accessing +`this.apiClient.apiKey` directly. Intentional (SSE streaming not supported by ApiClient) +but creates a maintenance risk if ApiClient's header logic changes. +- **Watch for**: new commands that bypass ApiClient for non-standard protocols — they + should document why. + +**Exit codes** +The convention is settled in `.claude/rules/commands.md`: **exit 2** for input validation +(`requireNumericId`, `requireUuid`, `parseJsonFlag`), **exit 1** for general errors, including the +empty-body guard above. +- **Watch for**: new commands using the wrong code. Check the rule before reporting one as wrong. + +**Hand-rolled pagination or sorting flags** +`src/lib/flags.ts` owns `paginationFlags` and `sortFlags` (page is 0-indexed with `min: 0`, +page-size `min: 1`), plus `addPagination`/`addSorting` for the query string. Declaring these +inline per command is how the defaults drifted apart in the first place. +- **Watch for**: a list command declaring its own `page`/`page-size`/`sort` flags, or declaring + pagination flags it then never sends as query params. + +--- + +## Per-agent mapping + +| Agent | Relevant pattern sections | +|---|---| +| **Correctness** | Guessed contract fields, unwrapped JSON.parse, double-parse, empty-body guard, exit codes | +| **Consistency** | Flag-rename sweep, hand-rolled pagination/sorting flags, double-parse, exit codes | +| **Security** | ask-genie bypasses ApiClient (direct apiKey access) | +| **Testing** | Missing `lastBody()` assertion on any command that sends a body; missing error-path tests for JSON.parse, resource IDs and empty bodies — each has a sweep under `test/validation/` whose final test asserts the table still covers every call site | diff --git a/.eslintrc.json b/.eslintrc.json new file mode 100644 index 0000000..fafab26 --- /dev/null +++ b/.eslintrc.json @@ -0,0 +1,9 @@ +{ + "extends": ["oclif", "oclif-typescript"], + "ignorePatterns": ["lib", "node_modules", "bin"], + "rules": { + "array-bracket-newline": "off", + "array-element-newline": "off", + "valid-jsdoc": "off" + } +} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000..18c8e84 --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,29 @@ +name: Tests +on: + workflow_call: + workflow_dispatch: + pull_request: + types: [opened, reopened, synchronize] + branches: [main] + +jobs: + Test: + runs-on: ubuntu-latest + steps: + - name: "Checkout" + uses: actions/checkout@v4 + - name: "Setup Node" + uses: actions/setup-node@v4 + with: + node-version: '22.x' + cache: 'npm' + - name: "Installing dependencies" + run: npm ci + - name: "Typecheck" + run: | + npx tsc -b + npx tsc -p tsconfig.test.json --noEmit + # Runs the linter first via the pretest hook, then the unit suite. + # The e2e suite is deliberately not run here: it needs a live API and a key. + - name: "Lint and test" + run: npm test diff --git a/.gitignore b/.gitignore index 1a7c1a1..a636745 100644 --- a/.gitignore +++ b/.gitignore @@ -4,5 +4,8 @@ node_modules/ oclif.manifest.json .DS_Store .agents/ -.claude/ +.claude/settings.json skills-lock.json + +# e2e undo log for mutations to shared resources +.e2e-restore.json diff --git a/.mocharc.e2e.yml b/.mocharc.e2e.yml new file mode 100644 index 0000000..432bb15 --- /dev/null +++ b/.mocharc.e2e.yml @@ -0,0 +1,11 @@ +# End-to-end suite: spawns the built CLI against a real API. +# Run with `npm run test:e2e`. See test/e2e/README.md for configuration. +# +# Deliberately NOT set: +# parallel — suites share one account's state +# retries — a re-run `it` would create its resources twice +node-option: + - loader=ts-node/esm +spec: test/e2e/**/*.e2e.ts +timeout: 180000 +slow: 5000 diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..49f5ee0 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,188 @@ +# Changelog + +## 1.0.0 — V2 API Migration + +**Breaking change**: The CLI now exclusively uses the Databox V2 API. All V1 API calls have been removed. This requires a Databox account with V2 API access. + +- **Removed** — `account data-sources` and `account datasets`. They were v1 spellings of `data-source list` and `dataset list` against the same endpoints; use those, with `--account-id` to target another account. +- **Activity log** nested under account — `activity-log list` now calls `/v2/account/activity-log` (was `/v2/activity-log`). The command name is unchanged. +- **New command** — `profile metadata-options` returns available departments and roles for profile metadata updates. + +### Migration Guide + +#### Authentication + +No changes to the authentication flow. API keys work the same way — `databox auth login` and the `DATABOX_API_KEY` environment variable continue to work as before. + +#### Command Changes — Where Did My Stuff Go? + +Every v0.x command is still supported in some form. Here's exactly where each one moved and what changed: + +| v0.x Command | v1.0 Equivalent | What Changed | +|---|---|---| +| `account list` | `account info` | **Renamed.** V2 returns your own account as a single object. To list accounts you manage, use `client list` (agency/client model). To access a specific account's resources, pass `--account-id` on any command. | +| `account data-sources ACCOUNTID` | `data-source list` | **Removed.** Use `data-source list`, with the global `--account-id` flag to target another account. | +| `account datasets ACCOUNTID` | `dataset list` | **Removed.** Use `dataset list`, with the global `--account-id` flag to target another account. `--type` filter removed — merged datasets are not in V2 (deferred to backlog). | +| `account timezones` | `account timezones` | No changes. | +| `data-source create` | `data-source create` | `--account-id` is now a global flag (works on all commands). `--key` is renamed to `--integration-key` for third-party integrations (e.g., Datadoo). | +| `data-source datasets ID` | `data-source datasets ID` | No user-facing changes. | +| `data-source delete ID` | `data-source delete ID` | No changes. | +| `dataset create` | `dataset create` | `--primary-keys` renamed to `--primary-key` (singular, still accepts multiple values). Schema column field `name` renamed to `columnId`. See schema example below. | +| `dataset get GUID` | `dataset get NUMERIC_ID` | **IDs are now numeric.** Use `dataset list` to find your dataset's numeric ID. | +| `dataset delete GUID` | `dataset delete NUMERIC_ID` | **IDs are now numeric.** | +| `dataset ingest GUID` | `dataset ingest NUMERIC_ID` | **IDs are now numeric.** | +| `dataset ingestion GUID ING_ID` | `dataset ingestion NUMERIC_ID ING_ID` | **Dataset ID is now numeric.** Ingestion ID unchanged. | +| `dataset ingestions GUID` | `dataset ingestions NUMERIC_ID` | **IDs are now numeric.** | +| `dataset purge GUID` | `dataset purge NUMERIC_ID` | **IDs are now numeric.** | +| `analyze ask-genie` | `analyze ask-genie` | No changes (uses separate agentic service). | + +#### New Global Flag + +| Flag | Env Var | Description | +|---|---|---| +| `--account-id` | `DATABOX_ACCOUNT_ID` | Target a specific account for multi-account access (agency/client model). Replaces the `ACCOUNTID` positional arg from v0.x. Works on all commands. | + +#### Schema Definition Change + +v0.x: +```bash +--schema '[{"name":"date","dataType":"datetime"},{"name":"value","dataType":"number"}]' +``` + +v1.0: +```bash +--schema '[{"columnId":"date","dataType":"datetime"},{"columnId":"value","dataType":"number"}]' +``` + +The `name` field was renamed to `columnId` to match the V2 API contract. + +#### Dataset ID Migration + +V1 used GUID identifiers for datasets (e.g., `a1b2c3d4-e5f6-...`). V2 uses numeric IDs (e.g., `12345`). The CLI now validates that dataset IDs are numeric and rejects non-numeric values with a clear error. + +To find the numeric ID for an existing dataset: +```bash +databox dataset list +``` + +#### Summary of Removed Features + +| Feature | Why | Alternative | +|---|---|---| +| Multi-account listing (`account list`) | V2 scopes to the caller's account | `client list` for managed accounts, `account info` for your own | +| `--type` filter on dataset listing | Merged datasets deferred to backlog | All datasets are regular datasets in V2 | +| GUID dataset IDs | Architectural decision — numeric IDs unify data sources and datasets | Use `dataset list` to find numeric IDs | +| `ACCOUNTID` positional arg | Replaced by header-based account scoping | `--account-id` flag (global, works everywhere) | + +### New Commands + +65+ new commands covering the full V2 API surface: + +#### Account +- `account info` — Show your account details +- `account update` — Update account name/settings +- `account usage` — Show usage statistics +- `account timezones` — List supported timezones + +#### Profile +- `profile info` — Show your profile +- `profile update` — Update your name or timezone + +#### Billing +- `billing info` — Show billing and plan details +- `billing invoices` — List invoices + +#### Users +- `user list` — List users in the account +- `user get` — Get user details +- `user invite` — Invite a new user +- `user update` — Update a user's role +- `user delete` — Remove a user + +#### Clients +- `client list` — List client accounts +- `client get` — Get client account details +- `client create` — Create a client account +- `client update` — Update a client account +- `client delete` — Delete a client account + +#### Connections +- `connection list` — List connections +- `connection get` — Get connection details +- `connection update` — Update a connection +- `connection delete` — Delete a connection +- `connection permissions` — Show permissions +- `connection set-permissions` — Update permissions + +#### Integrations +- `integration list` — Browse available integrations +- `integration get` — Get integration details + +#### Data Sources (new sub-commands) +- `data-source list` — List data sources with search and pagination +- `data-source get` — Get data source details +- `data-source update` — Update data source title +- `data-source set-timezone` — Set timezone +- `data-source sync-frequencies` — List available sync frequencies +- `data-source set-sync-frequency` — Set sync frequency +- `data-source permissions` — Show permissions +- `data-source set-permissions` — Update permissions +- `data-source purge` — Purge all data + +#### Datasets (new sub-commands) +- `dataset list` — List datasets with search and pagination +- `dataset update` — Update dataset title +- `dataset duplicate` — Duplicate a dataset +- `dataset data` — View dataset data +- `dataset schema` — View dataset schema +- `dataset set-timezone` — Set timezone +- `dataset sync-frequencies` — List available sync frequencies +- `dataset set-sync-frequency` — Set sync frequency +- `dataset sync-history` — View sync history +- `dataset permissions` — Show permissions +- `dataset set-permissions` — Update permissions +- `dataset metadata` — View metadata +- `dataset set-metadata` — Update metadata +- `dataset column-metadata` — View column metadata +- `dataset set-column-metadata` — Update column metadata +- `dataset verification` — View verification status +- `dataset set-verification` — Toggle verification +- `dataset modifications` — List modifications +- `dataset add-modification` — Add a modification +- `dataset update-modification` — Update a modification +- `dataset clear-modifications` — Clear all modifications +- `dataset preview-modification` — Preview a modification before applying +- `dataset modification-rules` — List available modification rules +- `dataset modification-formulas` — List available modification formulas +- `dataset ingestion-statistics` — View ingestion statistics +- `dataset sync-statistics` — View sync history statistics +- `dataset lineage` — Show dataset parents and children + +#### Metrics +- `metric list` — List metrics +- `metric get` — Get metric details +- `metric create` — Create a custom metric +- `metric update` — Update a metric +- `metric delete` — Delete a metric +- `metric data` — Load metric data +- `metric dimension-values` — Get dimension values +- `metric drilldown` — Get drilldown data +- `metric usages` — See where a metric is used +- `metric verification` — View verification status +- `metric set-verification` — Toggle verification + +#### Activity Log +- `activity-log list` — List activity log entries + +#### Databoards +- `databoard list` — List databoards +- `databoard metrics` — View metrics on a databoard + +### Unchanged + +- `auth login` — Authentication flow unchanged +- `auth validate` — Key validation unchanged +- `analyze ask-genie` — Genie AI integration unchanged (uses separate agentic service) +- `--json` flag — Works the same on all commands +- Config file location — `~/.config/databox-cli/config.json` unchanged +- `DATABOX_API_KEY` / `DATABOX_API_URL` env vars — Work the same diff --git a/README.md b/README.md index b016d92..058bd46 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # databox-cli -CLI for the [Databox](https://databox.com) public API. Manage accounts, data sources, datasets, push data, and analyze datasets with Genie AI — all from the terminal. +CLI for the [Databox](https://databox.com) V2 API. Manage accounts, data sources, datasets, metrics, connections, users, billing, and more — all from the terminal. ## Installation @@ -17,11 +17,21 @@ databox auth login # Verify your key works databox auth validate -# List your accounts -databox account list +# View your account +databox account info + +# List data sources +databox data-source list + +# Create a data source and dataset +databox data-source create --name "My Data Source" +databox dataset create --name "My Dataset" --data-source-id 12345 # Push data into a dataset -databox dataset ingest DATASET_ID --file data.json +databox dataset ingest 67890 --file data.json + +# List metrics +databox metric list ``` ## Authentication @@ -34,12 +44,34 @@ You can also pass the key inline: databox auth login --api-key YOUR_API_KEY ``` +## Global Flags + +| Flag | Env Var | Description | +|------|---------|-------------| +| `--json` | — | Output as JSON instead of table | +| `--api-key` | `DATABOX_API_KEY` | Override the stored API key | +| `--api-url` | `DATABOX_API_URL` | Override the API base URL | +| `--account-id` | `DATABOX_ACCOUNT_ID` | Target a specific account (for agency/client access) | + ## Output Formats By default, commands output human-readable tables. Add `--json` to any command for machine-readable JSON output: ```bash -databox account list --json +databox account info --json +databox data-source list --json +``` + +## Multi-Account Access + +For agency accounts managing client accounts, use the `--account-id` flag to scope commands to a specific client: + +```bash +# List your client accounts +databox client list + +# List data sources for a specific client +databox data-source list --account-id 12345 ``` ## Agent Skills @@ -51,9 +83,15 @@ This package includes shareable skills for AI agents (like [Claude Code](https:/ | Skill | Description | |-------|-------------| | `databox-auth` | Authentication setup and API key validation | -| `databox-accounts` | Account discovery, timezones, resource listing | -| `databox-data-sources` | Data source create, delete, and inspection | -| `databox-datasets` | Dataset CRUD, schema definition, data ingestion, monitoring | +| `databox-account` | Account info, usage, settings, timezones | +| `databox-data-sources` | Data source CRUD, timezone, sync, permissions, purge | +| `databox-datasets` | Dataset CRUD, schema, data ingestion, metadata, verification, modifications | +| `databox-metrics` | Metric CRUD, data loading, dimensions, drilldown, verification | +| `databox-users` | User invite, role management, removal | +| `databox-clients` | Client account management (agency model) | +| `databox-connections` | Connection management and permissions | +| `databox-integrations` | Browse available integration types | +| `databox-billing` | Billing info and invoices | | `databox-analyze` | Dataset analysis with Genie AI, conversational data Q&A | ### Install Skills @@ -68,116 +106,186 @@ Or install individual skills: ```bash npx skills add databox/databox-cli --skill databox-auth -npx skills add databox/databox-cli --skill databox-accounts +npx skills add databox/databox-cli --skill databox-account npx skills add databox/databox-cli --skill databox-data-sources npx skills add databox/databox-cli --skill databox-datasets +npx skills add databox/databox-cli --skill databox-metrics +npx skills add databox/databox-cli --skill databox-users +npx skills add databox/databox-cli --skill databox-clients +npx skills add databox/databox-cli --skill databox-connections +npx skills add databox/databox-cli --skill databox-integrations +npx skills add databox/databox-cli --skill databox-billing npx skills add databox/databox-cli --skill databox-analyze ``` -Once installed, Claude Code can manage your Databox resources directly — creating data sources, defining schemas, pushing data, monitoring ingestions, and analyzing datasets with Genie AI. +Once installed, Claude Code can manage your Databox resources directly — managing accounts, data sources, datasets, metrics, users, connections, billing, and analyzing data with Genie AI. + +## Changelog + +See the [changelog](https://github.com/databox/databox-cli/blob/main/CHANGELOG.md) for migration guides and version history. ## Commands -* [`databox account data-sources ACCOUNTID`](#databox-account-data-sources-accountid) -* [`databox account datasets ACCOUNTID`](#databox-account-datasets-accountid) -* [`databox account list`](#databox-account-list) +* [`databox account countries`](#databox-account-countries) +* [`databox account info`](#databox-account-info) +* [`databox account metadata-options`](#databox-account-metadata-options) * [`databox account timezones`](#databox-account-timezones) +* [`databox account update`](#databox-account-update) +* [`databox account usage`](#databox-account-usage) +* [`databox activity-log list`](#databox-activity-log-list) * [`databox analyze ask-genie DATASETID QUESTION`](#databox-analyze-ask-genie-datasetid-question) * [`databox auth login`](#databox-auth-login) * [`databox auth validate`](#databox-auth-validate) +* [`databox billing info`](#databox-billing-info) +* [`databox billing invoices`](#databox-billing-invoices) +* [`databox client create`](#databox-client-create) +* [`databox client delete CLIENTID`](#databox-client-delete-clientid) +* [`databox client get CLIENTID`](#databox-client-get-clientid) +* [`databox client list`](#databox-client-list) +* [`databox client update CLIENTID`](#databox-client-update-clientid) +* [`databox connection delete CONNECTIONID`](#databox-connection-delete-connectionid) +* [`databox connection get CONNECTIONID`](#databox-connection-get-connectionid) +* [`databox connection list`](#databox-connection-list) +* [`databox connection permissions CONNECTIONID`](#databox-connection-permissions-connectionid) +* [`databox connection set-permissions CONNECTIONID`](#databox-connection-set-permissions-connectionid) +* [`databox connection update CONNECTIONID`](#databox-connection-update-connectionid) * [`databox data-source create`](#databox-data-source-create) * [`databox data-source datasets DATASOURCEID`](#databox-data-source-datasets-datasourceid) * [`databox data-source delete DATASOURCEID`](#databox-data-source-delete-datasourceid) +* [`databox data-source get DATASOURCEID`](#databox-data-source-get-datasourceid) +* [`databox data-source list`](#databox-data-source-list) +* [`databox data-source permissions DATASOURCEID`](#databox-data-source-permissions-datasourceid) +* [`databox data-source purge DATASOURCEID`](#databox-data-source-purge-datasourceid) +* [`databox data-source set-permissions DATASOURCEID`](#databox-data-source-set-permissions-datasourceid) +* [`databox data-source set-sync-frequency DATASOURCEID`](#databox-data-source-set-sync-frequency-datasourceid) +* [`databox data-source set-timezone DATASOURCEID`](#databox-data-source-set-timezone-datasourceid) +* [`databox data-source sync-frequencies DATASOURCEID`](#databox-data-source-sync-frequencies-datasourceid) +* [`databox data-source update DATASOURCEID`](#databox-data-source-update-datasourceid) +* [`databox databoard list`](#databox-databoard-list) +* [`databox databoard metrics DATABOARDID`](#databox-databoard-metrics-databoardid) +* [`databox dataset add-modification DATASETID`](#databox-dataset-add-modification-datasetid) +* [`databox dataset clear-modifications DATASETID`](#databox-dataset-clear-modifications-datasetid) +* [`databox dataset column-metadata DATASETID`](#databox-dataset-column-metadata-datasetid) * [`databox dataset create`](#databox-dataset-create) +* [`databox dataset data DATASETID`](#databox-dataset-data-datasetid) * [`databox dataset delete DATASETID`](#databox-dataset-delete-datasetid) +* [`databox dataset duplicate DATASETID`](#databox-dataset-duplicate-datasetid) * [`databox dataset get DATASETID`](#databox-dataset-get-datasetid) * [`databox dataset ingest DATASETID`](#databox-dataset-ingest-datasetid) * [`databox dataset ingestion DATASETID INGESTIONID`](#databox-dataset-ingestion-datasetid-ingestionid) +* [`databox dataset ingestion-statistics DATASETID`](#databox-dataset-ingestion-statistics-datasetid) * [`databox dataset ingestions DATASETID`](#databox-dataset-ingestions-datasetid) +* [`databox dataset lineage DATASETID`](#databox-dataset-lineage-datasetid) +* [`databox dataset list`](#databox-dataset-list) +* [`databox dataset metadata DATASETID`](#databox-dataset-metadata-datasetid) +* [`databox dataset modification-formulas`](#databox-dataset-modification-formulas) +* [`databox dataset modification-rules`](#databox-dataset-modification-rules) +* [`databox dataset modifications DATASETID`](#databox-dataset-modifications-datasetid) +* [`databox dataset permissions DATASETID`](#databox-dataset-permissions-datasetid) +* [`databox dataset preview-modification DATASETID`](#databox-dataset-preview-modification-datasetid) * [`databox dataset purge DATASETID`](#databox-dataset-purge-datasetid) +* [`databox dataset schema DATASETID`](#databox-dataset-schema-datasetid) +* [`databox dataset set-column-metadata DATASETID`](#databox-dataset-set-column-metadata-datasetid) +* [`databox dataset set-metadata DATASETID`](#databox-dataset-set-metadata-datasetid) +* [`databox dataset set-permissions DATASETID`](#databox-dataset-set-permissions-datasetid) +* [`databox dataset set-sync-frequency DATASETID`](#databox-dataset-set-sync-frequency-datasetid) +* [`databox dataset set-timezone DATASETID`](#databox-dataset-set-timezone-datasetid) +* [`databox dataset set-verification DATASETID`](#databox-dataset-set-verification-datasetid) +* [`databox dataset sync-frequencies DATASETID`](#databox-dataset-sync-frequencies-datasetid) +* [`databox dataset sync-history DATASETID`](#databox-dataset-sync-history-datasetid) +* [`databox dataset sync-statistics DATASETID`](#databox-dataset-sync-statistics-datasetid) +* [`databox dataset update DATASETID`](#databox-dataset-update-datasetid) +* [`databox dataset update-modification DATASETID`](#databox-dataset-update-modification-datasetid) +* [`databox dataset verification DATASETID`](#databox-dataset-verification-datasetid) * [`databox help [COMMAND]`](#databox-help-command) - -## `databox account data-sources ACCOUNTID` - -List data sources for a specific account +* [`databox integration get INTEGRATIONID`](#databox-integration-get-integrationid) +* [`databox integration list`](#databox-integration-list) +* [`databox metric create`](#databox-metric-create) +* [`databox metric data`](#databox-metric-data) +* [`databox metric delete METRICID`](#databox-metric-delete-metricid) +* [`databox metric dimension-values`](#databox-metric-dimension-values) +* [`databox metric drilldown`](#databox-metric-drilldown) +* [`databox metric get METRICID`](#databox-metric-get-metricid) +* [`databox metric list`](#databox-metric-list) +* [`databox metric set-verification METRICID`](#databox-metric-set-verification-metricid) +* [`databox metric update METRICID`](#databox-metric-update-metricid) +* [`databox metric usages METRICID`](#databox-metric-usages-metricid) +* [`databox metric verification METRICID`](#databox-metric-verification-metricid) +* [`databox profile info`](#databox-profile-info) +* [`databox profile metadata-options`](#databox-profile-metadata-options) +* [`databox profile update`](#databox-profile-update) +* [`databox user delete USERID`](#databox-user-delete-userid) +* [`databox user get USERID`](#databox-user-get-userid) +* [`databox user invite`](#databox-user-invite) +* [`databox user list`](#databox-user-list) +* [`databox user update USERID`](#databox-user-update-userid) + +## `databox account countries` + +List available countries ``` USAGE - $ databox account data-sources ACCOUNTID [--json] - -ARGUMENTS - ACCOUNTID The account ID to list data sources for + $ databox account countries [--json] FLAGS --json Output as JSON DESCRIPTION - List data sources for a specific account + List available countries EXAMPLES - $ databox account data-sources 12345 + $ databox account countries - $ databox account data-sources 12345 --json + $ databox account countries --json ``` -_See code: [src/commands/account/data-sources.ts](https://github.com/databox/databox-cli/blob/v0.2.1/src/commands/account/data-sources.ts)_ +_See code: [src/commands/account/countries.ts](https://github.com/databox/databox-cli/blob/v1.0.0/src/commands/account/countries.ts)_ -## `databox account datasets ACCOUNTID` +## `databox account info` -List datasets for a specific account +Show your account details ``` USAGE - $ databox account datasets ACCOUNTID [--json] [--page ] [--page-size ] [--type - datasets|merged_datasets] - -ARGUMENTS - ACCOUNTID The account ID to list datasets for + $ databox account info [--json] FLAGS - --json Output as JSON - --page= Page number - --page-size= Number of items per page - --type=