From 08d30402a2ebbb8df742f20be153879943edce1b Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Wed, 23 Sep 2026 22:05:39 -0700 Subject: [PATCH 1/5] docs: clarify that uploads go through the S3-compatible data proxy The upload guide read as if data lived in AWS. Explain that the data proxy speaks the S3 API and may be backed by any object store, that AWS tools are only S3 clients, and that credentials come from Source. Fix stale s3://us-west-2.opendata.source.coop addresses in Option 2 and call out that Option 3 is the one path that is genuinely AWS. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/using-source/data-upload.md | 105 ++++++++++++++++++++++++------- 1 file changed, 81 insertions(+), 24 deletions(-) diff --git a/docs/using-source/data-upload.md b/docs/using-source/data-upload.md index bd30ff1..94c1ba1 100644 --- a/docs/using-source/data-upload.md +++ b/docs/using-source/data-upload.md @@ -5,19 +5,49 @@ slug: /data-upload sidebar_position: 2 --- -This guide explains how to deliver your data to Source Cooperative in a secure and simple way. -It is written for data providers and does not require deep Amazon Web Service (AWS) knowledge. +This guide explains how to deliver your data to Source Cooperative in a secure and simple way. +It is written for data providers and does not require an Amazon Web Services (AWS) account or deep cloud storage knowledge. If you do not see the option to upload (for example, Edit Mode or View Credentials on your product page under the lock icon), contact [hello@source.coop](mailto:hello@source.coop) to request upload access. --- +## How uploads work: the Source data proxy + +Unless you set up [direct bucket access](#option-3-longstanding-or-automated-access-advanced), +every upload goes through the **Source data proxy** at `https://data.source.coop`. +The proxy speaks the **S3 API**, the object storage protocol that Amazon S3 +introduced and that most storage tools now support. It is not itself an AWS service. + +Behind the proxy, a product's data may be stored with any of several object +storage providers: Amazon S3, Google Cloud Storage, Azure Blob Storage, +Cloudflare R2, and others. You don't need to know which one. You always +upload to the same proxy address, and the proxy writes to the right place. + +That is why this guide uses AWS tools: + +- **The AWS CLI and AWS SDKs (boto3, etc.) are used only as S3 clients.** You + point them at `https://data.source.coop` instead of AWS. Any S3-compatible + client that lets you set a custom endpoint URL will work. +- **Your upload credentials come from Source Cooperative, not AWS.** They use + the AWS credential format (access key ID, secret access key, session token) + because S3 clients expect it, but they are only valid at the Source data proxy. +- **Addresses are proxy addresses.** `s3://your-org/your-product/` means the + `your-product` product in the `your-org` account on Source Cooperative. It + is not an Amazon S3 bucket. Where your data is physically stored doesn't change it. + +The one exception is [Option 3](#option-3-longstanding-or-automated-access-advanced), +which writes straight to Source's Amazon S3 bucket with your own AWS identity. +In that section, "AWS" really does mean AWS. + +--- + ## The short version (what you need to do) You have two main ways to upload data: -1. **Upload directly in the Source Cooperative User Interface (UI)** (drag-and-drop or file selector), or -2. **Use temporary AWS credentials** to upload via the AWS Command Line Interface (CLI) or Software Development Kits (SDKs) +1. **Upload directly in the Source Cooperative User Interface (UI)** (drag-and-drop or file selector), or +2. **Use temporary Source credentials** to upload through the data proxy with an S3 client such as the AWS Command Line Interface (CLI) or an AWS Software Development Kit (SDK) For long‑term or automated access, contact the Source Cooperative team at `hello@source.coop`. @@ -25,7 +55,7 @@ For long‑term or automated access, contact the Source Cooperative team at `hel ## Option 1: Upload directly in the UI (Easiest) -If you prefer not to use AWS tools, you can upload files directly in the web interface. +If you prefer not to use command-line tools, you can upload files directly in the web interface. ### How to upload via the UI @@ -33,7 +63,7 @@ If you prefer not to use AWS tools, you can upload files directly in the web int 2. In the top-right corner of the Product Contents card, click on the lock icon to open the dropdown menu and enable edit mode by clicking on `Edit Mode`. 3. Either: - Drag‑and‑drop files to the `Product Contents` card - - Use `Upload Files` or `Upload Directory` options in dropdown menu to upload files via operating system's file selectory + - Use `Upload Files` or `Upload Directory` options in dropdown menu to upload files via operating system's file selector Your files will be uploaded automatically to your product. @@ -45,13 +75,14 @@ This option is ideal for: --- -## Option 2: Upload using temporary AWS credentials (recommended for larger uploads) +## Option 2: Upload through the data proxy with temporary credentials (recommended for larger uploads) -For larger uploads, scripting, or programmatic access, use temporary AWS credentials. +For larger uploads, scripting, or programmatic access, use temporary credentials +issued by Source Cooperative with an S3 client pointed at the data proxy. ### Get credentials with the Source CLI (recommended) -The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you and provides temporary AWS credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS profile, the AWS CLI and SDKs refresh credentials for you. +The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you with Source Cooperative and provides temporary credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS CLI profile, the AWS CLI and SDKs refresh credentials for you. **1. Install the CLI** @@ -65,9 +96,9 @@ source-coop login This opens your browser to authenticate and caches short‑lived credentials in your OS keyring. -**3. Configure an AWS profile** +**3. Configure an AWS CLI profile** -Add a profile to `~/.aws/config` that calls the CLI to fetch credentials on demand: +Add a profile to `~/.aws/config` that calls the Source CLI to fetch credentials on demand and sends requests to the data proxy: ```ini [profile source-coop] @@ -75,7 +106,9 @@ credential_process = source-coop creds endpoint_url = https://data.source.coop ``` -**4. Use standard AWS commands with the profile** +The `endpoint_url` line is what sends requests to the Source data proxy instead of AWS. + +**4. Use standard S3 commands with the profile** Pass the profile per command with `--profile`: @@ -112,12 +145,17 @@ The AWS CLI will refresh credentials automatically; re-run `source-coop login` w You will also see: - `Expiration`: the expiration time of the credentials (a specific date and time) -- `Bucket`: the bucket name (`your-org`) +- `Bucket`: the bucket name on the data proxy, which is your account ID (`your-org`) - `Prefix`: the prefix (folder) you are allowed to write to (e.g. `your-product/`) --- -## What the credentials look ilke +## What the credentials look like + +These are Source Cooperative credentials in the AWS credential format. They +only work against the data proxy, so always set the endpoint to +`https://data.source.coop`. The region value is required by S3 clients; it does +not say where your data is stored. ### For SDK clients (boto3, AWS SDKs, etc.) @@ -130,6 +168,15 @@ You will also see: } ``` +With boto3, pass the endpoint when creating the client: + +```python +import boto3 + +s3 = boto3.client("s3", endpoint_url="https://data.source.coop", **credentials) +s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") +``` + --- ### For terminal / shell usage @@ -139,6 +186,7 @@ export AWS_ACCESS_KEY_ID="ASIA..." export AWS_SECRET_ACCESS_KEY="pwEV..." export AWS_SESSION_TOKEN="IQoJ..." export AWS_DEFAULT_REGION="us-west-2" +export AWS_ENDPOINT_URL="https://data.source.coop" # add this: send requests to the data proxy ``` These credentials are: @@ -151,17 +199,13 @@ These credentials are: ## Where to upload your data -You may upload only to the provided prefix, for example: +You may upload only to your product's prefix on the data proxy, for example: ``` -s3://us-west-2.opendata.source.coop/your-org/your-product/ +s3://your-org/your-product/ ``` -or if you are not uploading under an organization: - -``` -s3://us-west-2.opendata.source.coop/your-product/ -``` +This matches your product page URL, `https://source.coop/your-org/your-product`. You may upload: @@ -176,15 +220,17 @@ You may not upload outside this path. ## Example: Upload using the AWS CLI ```bash -aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv +aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --endpoint-url https://data.source.coop ``` Or upload a full directory: ```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ +aws s3 sync ./data s3://your-org/your-product/ --endpoint-url https://data.source.coop ``` +`--endpoint-url` isn't needed if you set `AWS_ENDPOINT_URL` or use the `source-coop` profile above. + --- ## Option 3: Long‑standing or automated access (ADVANCED) @@ -197,6 +243,17 @@ If you need: You can use your own IAM role to write to the Source Cooperative bucket. +:::info This option really is AWS + +Unlike Options 1 and 2, this option bypasses the data proxy. You write +directly to Source Cooperative's Amazon S3 bucket (`us-west-2.opendata.source.coop`) +using an identity in **your own AWS account**. It only applies to products stored +in that bucket. If your product is stored with another provider, or you don't have +an AWS account, use [Option 2](#option-2-upload-through-the-data-proxy-with-temporary-credentials-recommended-for-larger-uploads) +or contact us. + +::: + ### How this works - You create an IAM role (or use an existing IAM user/account) in **your own** AWS account @@ -408,7 +465,7 @@ This protects both you and Source Cooperative. Please do not: -- Request permanent AWS access keys +- Request permanent access keys - Ask for full bucket access - Upload outside your assigned prefix - Reuse expired temporary credentials From d1070215c8ce15f0f129cf6a73b57c9d9566bbf0 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Thu, 24 Sep 2026 07:35:12 -0700 Subject: [PATCH 2/5] acknowledge other AWS buckets Clarified the S3 bucket examples for data upload. --- docs/using-source/data-upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/using-source/data-upload.md b/docs/using-source/data-upload.md index 94c1ba1..747944e 100644 --- a/docs/using-source/data-upload.md +++ b/docs/using-source/data-upload.md @@ -246,7 +246,7 @@ You can use your own IAM role to write to the Source Cooperative bucket. :::info This option really is AWS Unlike Options 1 and 2, this option bypasses the data proxy. You write -directly to Source Cooperative's Amazon S3 bucket (`us-west-2.opendata.source.coop`) +directly to Source Cooperative's Amazon S3 bucket (e.g. `us-west-2.opendata.source.coop`, `eu-west-1.opendata.source.coop`) using an identity in **your own AWS account**. It only applies to products stored in that bucket. If your product is stored with another provider, or you don't have an AWS account, use [Option 2](#option-2-upload-through-the-data-proxy-with-temporary-credentials-recommended-for-larger-uploads) From 29cd10e73e6860ab5f8e57eb55e4b8e56fea9b1f Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Thu, 24 Sep 2026 07:37:19 -0700 Subject: [PATCH 3/5] docs: drop bucket-owner-full-control ACL requirement Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/using-source/data-upload.md | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/docs/using-source/data-upload.md b/docs/using-source/data-upload.md index 747944e..610230a 100644 --- a/docs/using-source/data-upload.md +++ b/docs/using-source/data-upload.md @@ -258,7 +258,6 @@ or contact us. - You create an IAM role (or use an existing IAM user/account) in **your own** AWS account - You send us its ARN, and we grant it write access to your account's prefix in the Source Cooperative bucket -- You upload with `--acl bucket-owner-full-control` so Source Cooperative owns the objects No credentials are shared, and no role chaining is required. @@ -391,11 +390,10 @@ We will add the ARN to the bucket policy and confirm when it is active. ### Step 3: Upload using that identity -Once we confirm, upload with credentials for that role, user, or account. **You must set `bucket-owner-full-control`** so Source Cooperative fully owns and can manage the uploaded objects: +Once we confirm, upload with credentials for that role, user, or account: ```bash -aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv \ - --acl bucket-owner-full-control +aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv ``` Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) need no assume-role step — the SDK picks up the role automatically. @@ -404,8 +402,7 @@ Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) Uploading a directory ```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ \ - --acl bucket-owner-full-control +aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ ``` @@ -424,7 +421,6 @@ region = us-west-2 ```bash aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ \ - --acl bucket-owner-full-control \ --profile source-coop-upload ``` @@ -441,7 +437,6 @@ s3.upload_file( "mydata.csv", "us-west-2.opendata.source.coop", "your-org/your-product/mydata.csv", - ExtraArgs={"ACL": "bucket-owner-full-control"}, ) ``` @@ -469,7 +464,6 @@ Please do not: - Ask for full bucket access - Upload outside your assigned prefix - Reuse expired temporary credentials -- Upload without `bucket-owner-full-control` when using your own role --- From 227951f5fae8a3d5f004480971d2c2d622f83aa7 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Thu, 24 Sep 2026 20:52:11 -0700 Subject: [PATCH 4/5] Apply suggestion from @alukach --- docs/using-source/data-upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/using-source/data-upload.md b/docs/using-source/data-upload.md index 610230a..be31426 100644 --- a/docs/using-source/data-upload.md +++ b/docs/using-source/data-upload.md @@ -82,7 +82,7 @@ issued by Source Cooperative with an S3 client pointed at the data proxy. ### Get credentials with the Source CLI (recommended) -The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you with Source Cooperative and provides temporary credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS CLI profile, the AWS CLI and SDKs refresh credentials for you. +The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you with Source Cooperative and provides temporary credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS CLI profile, the AWS CLI and SDKs refresh credentials for you, and you'll only need to re-authenticate every 30 days. **1. Install the CLI** From 0eb2eab572c471e03b18f4466050c6963216688b Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Thu, 24 Sep 2026 20:54:25 -0700 Subject: [PATCH 5/5] Clarify --- docs/using-source/data-upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/using-source/data-upload.md b/docs/using-source/data-upload.md index be31426..32f57e9 100644 --- a/docs/using-source/data-upload.md +++ b/docs/using-source/data-upload.md @@ -343,7 +343,7 @@ in-progress uploads across the *whole* bucket. AWS does not support the `s3:prefix` condition key on it, so it cannot be limited to your data, and no upload path needs it — incomplete uploads are cleaned up automatically after 7 days. Only `aws s3api list-multipart-uploads` requires it; contact us if you -have a workflow that does. +have a workflow that requires this policy. :::