Use bounded server-side GCS copies for Core relocation tables

[Nils-Fedrigo]

Description

Replace per-data-source Storage Transfer Service job creation for Core table files with bounded server-side Cloud Storage copies. Follow-up to #32789 and #32790: skipping empty prefixes alone does not reduce create-quota usage enough for the ongoing relocation.

• Keep startTransferCoreTableFiles and the existing workflow command sequence unchanged. New invocations return null only after all listed objects have been copied and verified (or a successful empty listing). The existing null completion path is already deployed.
• Existing transferJobs/... handles recorded in Temporal history continue to be polled by the unchanged STS implementation. Front public/private file transfers remain on STS.
• Enumerate all pages via getAllFilesByPrefix; preserve relative names under the source-to-destination Core ID mapping.
• At most 5 concurrent object copy/verification tasks per activity, not a global limit. Reuse FileStorage.copyFile for server-side copy/rewrite and its retry logic. No CSV payload is downloaded into the worker.
• Pin the source generation from listing; require CRC32C and size; verify destination contents after each copy. Already matching objects are skipped on retries. A differing destination is overwritten, so destination write/delete permission is required.
• Wait for every in-flight sibling in a batch to settle before surfacing failures. Permission/listing/verification errors fail closed and never mark relocation complete.
• Stop starting new work after eight minutes within the existing ten-minute activity timeout. Completed objects serve as retry checkpoints through metadata verification. Observe activity cancellation between operations.
• Emit start/completion logs with object counts, verified-copy counts and elapsed time.

Tests

• Added helper regression tests for path mapping, generation pinning, checksum and size verification, retry skipping, mismatched destinations, missing metadata, permission failures, concurrency limits, draining siblings on failure and cancellation/budget checks.
• Updated activity tests for paginated enumeration, direct-copy completion/failure and legacy STS polling compatibility; retained environment-specific credential tests.
• Executed: 12 standalone Node assertion checks against the actual new helper, including partial-failure retry and max concurrency 5 across 13 objects; all passed. These are isolated mocked-storage smoke checks, not real GCS tests.
• Executed: isolated strict TypeScript check of copy_core_tables.ts; passed.
• Executed: Oxlint's default rule set on all four changed files (not the full repository lint configuration); zero warnings/errors.
• Executed: git diff --check; passed. Reviewed the complete local diff after a shallow main clone.
• Full repository Vitest/typecheck/formatter and live GCS/Temporal integration tests have not run in the Computer, which has no installed project dependencies. Require CI and owner review before rollout.

Risk

IAM prerequisite: existing source Object Viewer access is sufficient for reading, but the US worker must additionally have destination object read/create/delete access. The earlier read-only infra PR dust-tt/dust-infra#1116 does not provide that cross-region write access. Without it, the new activity will fail/retry before completion. No IAM changes are made by this PR.

Consistency: requires the relocation maintenance/write freeze on both source and destination. Source generations are pinned within each attempt; there is no durable manifest across attempts, so source writes must remain frozen. No source objects are deleted. Verified retries compare contents (CRC32C and size), not arbitrary metadata equality; new copies use the existing SDK copy behavior.

Timeout/size limits: the existing activity timeout remains ten minutes to avoid changing running workflow commands. The eight-minute guard does not abort an individual long-running GCS rewrite or listing. Very large objects may still hit the timeout, and very large prefixes may spend too long listing or checking already copied objects. Listing is paginated but accumulates object metadata in memory. This is a bounded-copy emergency path, not a general streaming bulk-migration engine. Validate representative large sources before broad rollout; do not claim deadline guarantees from this PR.

Concurrency: the limit is per activity, so aggregate copy traffic scales with activity slots/replicas and active data sources. Start at current worker capacity and observe GCS and Core health before scaling further. SDK copies already in flight can finish after a worker/activity timeout; immutable source data and destination verification make retries convergent.

Compatibility/rollback: no workflow resets or command changes. getBucketInstance caches by bucket name, so deploy fresh worker processes. Rollback must retain the existing null handling from #32789. Rolling back the copying activity returns future work to STS and consumes its create quota again. Coordinate rollback/grant removal with in-flight activities.

Deploy Plan

1. An authorized owner grants the source worker Object User (or equivalent scoped custom permissions) on the destination tables bucket. For the current US-to-EU relocation:

gcloud storage buckets add-iam-policy-binding gs://bkt-prj-dust-europe-west1-tables --member="serviceAccount:gcs-region-relocation-sa@or1g1n-186209.iam.gserviceaccount.com" --role="roles/storage.objectUser" --condition=None --format=none

This changes IAM and must be intentionally applied/reconciled in Terraform. It permits destination object reads/writes/deletes, not just listing. Other relocation directions require their own grants; this activity change applies to new table-copy activities generally.

2. Keep source and destination writes frozen. Validate an approved representative server-side copy from the source worker identity, including destination verification and a larger table. Do not use personal credentials as evidence of worker access.
3. After CI and review, deploy fresh source relocation workers. Keep the existing Temporal execution/history; do not reset or restart relocation from the beginning.
4. Confirm [GCS Copy] Table file copies completed and verified. and advancing Core batch cursors. Existing STS jobs may still run/poll; new table-copy activities should not create STS jobs.
5. Monitor copy durations, object counts, GCS failures, timeout retries and overall remaining work. Preserve time for Apps and cutover. Revisit oversized-source handling if activities repeatedly time out.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated
playground	Ignored	Preview	Sep 20, 2026 11:56am UTC
storybook	Ignored	Preview	Sep 20, 2026 11:56am UTC

[github-actions]

Warnings

⚠️

Temporal workflow/activity files have been modified: - front/temporal/relocation/activities/source_region/front/file_storage.ts **IMPORTANT**: Renaming activity functions can cause non-deterministic errors in Temporal workflows. Running workflows may still reference the old activity names, which will cause failures when they try to execute. Best practices: - **DO NOT** rename existing activity functions - **DO** create new activities with new names if you need different behavior - **DO** use activity versioning patterns if you must change activity signatures - **DO** ensure all running workflows complete before removing old activities If you're only adding new activities or fixing bugs within existing ones, this warning can be ignored.

Generated by 🚫 dangerJS against 497fc01