[Data] Add organization-wide import/export, backups, and scheduled warehouse/object-storage delivery

[rahuliitk]

Parent epic: #81

Priority: P2 — enterprise portability and scale

Problem and current baseline

QuickVoice supports feature-specific data movement such as knowledge-base uploads and campaign CSV/XLSX ingestion, along with call/extraction views. It does not yet provide a complete organization-wide, versioned export, scheduled downstream delivery, verified backup, or safe restore/import workflow. Customers cannot independently archive configuration and permitted business data, recover an organization, or feed a warehouse without custom database access.

Outcome

Authorized customers can request full or selective portable exports, schedule policy-filtered deliveries to their own storage/warehouse, verify completeness with manifests/checksums, and restore supported configuration/data into a clean or existing organization through previewed, audited conflict handling.

Requirements

• Define an export inventory and dependency graph for organization metadata, users/roles references, agents/versions/workflows, tools/MCP/connectors metadata, knowledge-source metadata and optionally source files, phone-number configuration references, contacts/consent/suppressions, campaigns, calls, recordings/transcripts, extracted data, evaluations, analytics events/aggregates, audit logs, templates, and billing/usage references where legally/exportably appropriate.
• Support full and selective exports by resource type, date range, project/environment, agent/campaign, and data class, with explicit inclusion estimates and policy exclusions before execution.
• Produce a versioned manifest describing schema/export version, organization, filters, creation time, watermark, consistency model, file list, record counts, byte sizes, checksums, relationships, redactions, omissions/warnings, encryption, and restore compatibility.
• Use open documented formats: JSON/JSONL and CSV/Parquet where suitable, with UTC timestamps, stable IDs, units/currency, schema files, and no dependence on internal database dumps for portability.
• Run exports/imports/restores as asynchronous idempotent jobs with progress, cancellation, bounded retries, expiration, resumable multipart transfer, and signed short-lived download/upload access.
• Encrypt artifacts in transit and at rest; support customer-managed destination credentials via write-only secret references and optional customer-managed encryption key design where enterprise policy permits.
• Schedule incremental/full delivery to S3-compatible object storage and selected warehouse destinations using checkpoint/watermark semantics, partitioning, schema evolution, delivery history, alerting, retry, and backfill.
• Implement import/restore validation before mutation: authenticate/decrypt, verify manifest/checksum/schema, scan/size limits, dependency order, external/provider prerequisites, ID mapping, quota/cost estimate, policy conflicts, and dry-run report.
• Define per-resource conflict modes such as fail, skip, create-new/remap, or update when supported. Destructive replacement is never the default; partial restore must be resumable or compensatable and visibly reported.
• Exclude plaintext credentials, API keys, OAuth tokens, signing secrets, and unrecoverable provider secrets. Export only secret-reference metadata; restore requires reauthorization or explicit separately encrypted mechanism approved by security design.
• Apply retention, deletion, legal hold, consent, zero-PII, recording/transcript access, data residency, and role policy at job start and artifact access time. Audit request, approval, generation, access/download, delivery, cancellation, import, conflict decision, and restore.
• Document self-hosted backup/restore procedures separately from portable organization export while sharing validation/manifests where possible.

Interface and data implications

• Define ExportJob, ExportSelection, Artifact/Part, ExportManifest, Schedule, Destination, DeliveryAttempt, ImportJob, ValidationReport, Conflict, IdMap, RestoreCheckpoint, and RestoreReport resources under API v1.
• Version resource schemas independently from archive layout and publish migration readers for supported prior versions.
• Emit signed job/delivery events and expose stable status/error codes; all listing/download/restore APIs are cursor-paginated and organization/role scoped.

UX and accessibility

Provide a selection and impact summary, estimated size/time/cost, protected-data warnings, destination test, job progress, manifest/omission summary, delivery history, dry-run conflicts, explicit restore confirmation, and downloadable failure report. Long tables, progress, and dialogs must be keyboard/screen-reader accessible; state cannot rely on color.

Security, privacy, provider, and cost boundaries

Require elevated permission and optional approval for full exports/restores. Use short-lived scoped URLs, encryption, malware/content checks for imports, tenant isolation, destination SSRF/egress controls, and immutable audit logs. Bound artifact size, concurrency, retention, destination frequency, warehouse load, and egress cost; show estimates without presenting them as invoices.

Failure modes and backward compatibility

Handle policy changes mid-job, expired URLs, checksum mismatch, corrupt/truncated archive, insufficient quota, destination outage/429, schema drift, unsupported version, duplicate/replayed job, missing dependencies, ID collision, partial delivery, partial restore, and revoked credentials. Existing feature-specific upload/download flows remain supported and can adopt shared job primitives without changing accepted formats unexpectedly.

Dependencies

• Stable API/jobs/idempotency in [API] Stabilize public API v1 with scoped keys, idempotency, pagination, rate limits, and OpenAPI contracts #108 and delivery events in [Events] Add signed webhooks, retries, replay, dead-letter handling, and streaming subscriptions #110.
• Canonical schemas/events in [Analytics] Define the canonical event, metric, cost, latency, quality, and outcome model #90.
• Connector destination/auth primitives in [Integrations] Build a connector framework and priority CRM, calendar, helpdesk, commerce, and automation connectors #118.
• Enterprise roles, retention, residency, legal hold, secrets, and audit controls.

Out of scope

• Exporting plaintext credentials or reconstructing third-party access without reauthorization.
• Guaranteed point-in-time recovery for external telephony/model/storage providers.
• Treating a portable export as a substitute for operator disaster-recovery backups.

Acceptance criteria

• An authorized user can dry-run and generate a selective or full export with a versioned manifest, counts, checksums, relationships, and explicit omissions.
• The same filtered fixture produces consistent documented records across API and export, subject to a declared watermark/consistency model.
• Scheduled object-storage delivery supports checkpointed incremental runs, retries, backfill, schema/version metadata, and visible history.
• A clean organization can validate and restore every supported portable resource with deterministic ID mapping and recorded external prerequisites.
• Existing-organization restore previews all conflicts and never destructively replaces data by default.
• No artifact or log contains retrievable API/OAuth/provider secret material; download/destination access is short-lived/scoped and audited.
• Corruption, partial failure, cancellation, and unsupported versions produce a resumable/compensatable state and a machine-readable report.

Tests

• Golden archive/manifest/schema, record-count/checksum, relationship, date/filter, watermark, and prior-version reader tests.
• Tenant/role/approval, zero-PII, redaction, consent, retention/deletion/legal-hold, secret exclusion, and signed-URL tests.
• Multipart resume, cancel/retry/idempotency, artifact expiry, size/quota, destination outage/429, incremental checkpoint, and backfill tests.
• Import malware/zip-bomb/path traversal, corrupt/checksum mismatch, unsupported schema, missing dependency, ID collision, dry-run, and conflict-mode tests.
• Clean restore, existing-org partial restore, rollback/compensation, reauthorization, and external-prerequisite tests.
• Keyboard, screen-reader, focus, progress announcement, non-color status, and destructive-confirmation checks.