Feat/databox v2 support

[bwiz]

Migrates the CLI to the Databox V2 API and brings every command into line with the
ingestion-api contracts. Breaking: the CLI no longer speaks V1 at all.

What's in it

V2 migration — all endpoints, the {data, requestId, status} envelope, 0-based pagination,
numeric resource IDs, and --account-id promoted to a global flag. Six dataset commands added for
full V2 coverage; activity-log moved under /v2/account/activity-log.

Contract sync — ingestion-api's C# contracts are the source of truth, and a mechanical audit
against them fixed ~24 commands: request bodies that sent field names the API ignores
(interval for syncInterval, status for isVerified, tags for synonyms), response
readers that assumed the wrong envelope, and six permanently blank table columns. Commands now
expose every field of their request contract and every query parameter their controller declares —
roughly 50 new flags — and --json returns what the endpoint returned rather than a hand-picked
subset.

End-to-end suite (test/e2e/, npm run test:e2e) — spawns the built binary against a real
environment and asserts exit codes and output. Everything goes through the CLI, so the suite cannot
drift from the client it tests. This is what found the contract bugs above; none of them were
visible to the mocked unit suite, because each mock encoded the same guess as the code it tested.
Defaults to develop6, takes DATABOX_E2E_ENV / DATABOX_E2E_API_URL / DATABOX_E2E_API_KEY, and
refuses to touch production without DATABOX_E2E_ALLOW_PROD=1. Mutations to shared resources are
recorded in a durable undo log so an interrupted run can be unwound with
npm run test:e2e:cleanup.

Review fixes — request timeouts (30s, 300s for uploads) where there were none, UUID validation
and path encoding on dataset ingestion, --access-level constrained to the values the API
accepts, pagination flags on dataset preview-modification actually sent, the 9 missing
oclif.topics entries, empty-2xx-body handling, and credential hygiene: the config file is written
0600, the unit suite no longer writes to the developer's real ~/.config/databox-cli/config.json,
and a caller-supplied header can no longer overwrite x-api-key.

Shared helpers — parseJsonFlag, requireUuid, paginationFlags/sortFlags, replacing
patterns that had been copied across 17 commands and had already drifted.

Breaking changes

• V1 is gone. Requires an account with V2 API access.
• account data-sources and account datasets are removed. They were v1 spellings of
  data-source list and dataset list against the same endpoints; use those with --account-id.
• Numeric resource IDs, and several renamed flags (--primary-keys→--primary-key,
  --key→--integration-key, --tags→--synonyms).

Full command-by-command mapping in the CHANGELOG migration guide.

Testing

• 301 unit tests, including sweeps for malformed JSON, invalid resource IDs and empty update
  bodies whose final test walks src/commands to assert the table still covers every call site.
• E2E green against develop6, with one exception: the dataset ingestion detail assertion expects
  the fields the V2 contract declares, and stays red until the ingestion-api fix for that is
  deployed (see below).
• Three ingestion-api bugs this surfaced: dataset list applied the dataSourceId filter after
  paging, and dataset duplicate surfaced an opaque upstream refusal — both merged (#48). The
  third, GetIngestion returning less than a list row, is on
  fix/v2-ingestion-detail-fields.

[AndreLei]

http://markdown-server.prod/view/d7b1bc6b-2c8b-4ae9-9222-9bd8afedc9c4

We need to address the above issues and improvements

[bwiz]

Thanks for this — the report was genuinely useful, and the blocker category it identified
(fields guessed from the endpoint name rather than read from the contract) turned out to be the
right diagnosis of the whole class of bug here.

One thing worth saying up front: the review was written before the contract-sync commits, so
B2–B5 were already fixed by the time it landed. I verified each one against the branch rather
than assuming:

• B2 (6 dead commands) — syncInterval, isVerified, {metrics:[…]} + dimensionValues,
  sharedWithClients all send the contract's field names.
• B3 (4 crashing renders) — sync-frequencies reads the bare array, column-metadata reads
  .items, modifications renders an array.
• B4 (timestamp) — gone; both commands render startedAt/finishedAt/duration/user.
• B5 (6 blank columns) — parentDataSourceId, createdAt/user, finishedAt populated; no
  phantom columns.

B1 was fixed in code but not in the docs. That's now swept: README regenerated, both skill
files, and the CHANGELOG line that still promised --key.

Everything else is addressed — warnings and all 14 suggestions:

W1	requireUuid + encodeURIComponent on dataset ingestion; examples use a real UUID
W2	--access-level constrained to the values the API accepts; specific_users was never one
W3	preview-modification actually sends --page/--page-size
W4	9 missing oclif.topics entries
W5	account datasets / account data-sources removed — v1 spellings of commands that already exist. Breaking, and called out in the CHANGELOG
W6	request timeouts (30s; 300s for uploads) and an idle timeout on ask-genie's reader
W7, S13	validation sweeps + list output-contract tests; 203 → 301 unit tests
S1, S2	empty-2xx-body guard; dataset ingest preflights array shape and the size limits
S5, S6	paginationFlags/sortFlags and parseJsonFlag, replacing 17 copies each
S8, S9, S12	showPagination edge cases, the null .join crash, exclusive on metric data, --managed-by-id 0, Errors: null
S11	config written 0600; the unit suite no longer writes the developer's real config; a caller header can no longer overwrite x-api-key; --account-id validated
S14	PR description written

Three places I deliberately did not follow the report:

1. B1's --data-source-id claim is incorrect. It says the flag was renamed to --source-id.
   It wasn't — --data-source-id is still live on dataset create, dataset list and
   metric data, while --source-id is what metric list and metric dimension-values use.
   Two different flags, not one rename; the README's references are correct, so I left them alone.
2. dataset update's empty-body guard stays exit 1. .claude/rules/commands.md specifies
   exit 1 for a well-formed command with nothing to do, and exit 2 for input validation. Happy to
   change the convention, but not in one command only.
3. S4 — added response interfaces where they prevent a real mistake, not to every call site.

S3 (--idempotency-key) is not done — it's a feature rather than a fix, so I left it out
rather than expanding the PR further. Say the word if you want it in.

Two things the review led to that were API bugs, both now merged in ingestion-api (#48):
dataset list applied the dataSourceId filter after paging (so it silently dropped matches on
other pages and reported the unfiltered total), and dataset duplicate surfaced an opaque upstream
refusal instead of explaining that API-created datasets can't be duplicated yet.

On B4 — you were right that the CLI side was only half the story. GetIngestion returned less than
a single list row because it read the ingest through the V1 contract, which has no
startedAt/finishedAt/duration/user at all, and left metrics and errors unpopulated too.
Fixed on fix/v2-ingestion-detail-fields: it now reads account-service directly, as
ListIngestions already did, and both views map through one shared helper so a detail response
can't end up thinner than a list row again.

CodeQL's js/clear-text-logging is green. The first pass at it was a rename that stopped the
heuristic matching without changing anything real; the fix that's in now separates the printable
target from the credential so the banner builder never has the key in scope, with tests that hold
independently of the scanner.