Skip to content

docs: clarify that uploads go through the S3-compatible data proxy #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alukach merged 5 commits into from
Sep 25, 2026
Merged

docs: clarify that uploads go through the S3-compatible data proxy #35

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
08d3040
docs: clarify that uploads go through the S3-compatible data proxy
alukach Sep 24, 2026
d107021
acknowledge other AWS buckets
alukach Sep 24, 2026
29cd10e
docs: drop bucket-owner-full-control ACL requirement
alukach Sep 24, 2026
227951f
Apply suggestion from @alukach
alukach Sep 25, 2026
0eb2eab
Clarify
alukach Sep 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 85 additions & 34 deletions docs/using-source/data-upload.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,35 +5,65 @@ slug: /data-upload
sidebar_position: 2
---

This guide explains how to deliver your data to Source Cooperative in a secure and simple way.
It is written for data providers and does not require deep Amazon Web Service (AWS) knowledge.
This guide explains how to deliver your data to Source Cooperative in a secure and simple way.
It is written for data providers and does not require an Amazon Web Services (AWS) account or deep cloud storage knowledge.

If you do not see the option to upload (for example, Edit Mode or View Credentials on your product page under the lock icon), contact [hello@source.coop](mailto:hello@source.coop) to request upload access.

---

## How uploads work: the Source data proxy

Unless you set up [direct bucket access](#option-3-longstanding-or-automated-access-advanced),
every upload goes through the **Source data proxy** at `https://data.source.coop`.
The proxy speaks the **S3 API**, the object storage protocol that Amazon S3
introduced and that most storage tools now support. It is not itself an AWS service.

Behind the proxy, a product's data may be stored with any of several object
storage providers: Amazon S3, Google Cloud Storage, Azure Blob Storage,
Cloudflare R2, and others. You don't need to know which one. You always
upload to the same proxy address, and the proxy writes to the right place.

That is why this guide uses AWS tools:

- **The AWS CLI and AWS SDKs (boto3, etc.) are used only as S3 clients.** You
point them at `https://data.source.coop` instead of AWS. Any S3-compatible
client that lets you set a custom endpoint URL will work.
- **Your upload credentials come from Source Cooperative, not AWS.** They use
the AWS credential format (access key ID, secret access key, session token)
because S3 clients expect it, but they are only valid at the Source data proxy.
- **Addresses are proxy addresses.** `s3://your-org/your-product/` means the
`your-product` product in the `your-org` account on Source Cooperative. It
is not an Amazon S3 bucket. Where your data is physically stored doesn't change it.

The one exception is [Option 3](#option-3-longstanding-or-automated-access-advanced),
which writes straight to Source's Amazon S3 bucket with your own AWS identity.
In that section, "AWS" really does mean AWS.

---

## The short version (what you need to do)

You have two main ways to upload data:

1. **Upload directly in the Source Cooperative User Interface (UI)** (drag-and-drop or file selector), or
2. **Use temporary AWS credentials** to upload via the AWS Command Line Interface (CLI) or Software Development Kits (SDKs)
1. **Upload directly in the Source Cooperative User Interface (UI)** (drag-and-drop or file selector), or
2. **Use temporary Source credentials** to upload through the data proxy with an S3 client such as the AWS Command Line Interface (CLI) or an AWS Software Development Kit (SDK)

For long‑term or automated access, contact the Source Cooperative team at `hello@source.coop`.

---

## Option 1: Upload directly in the UI (Easiest)

If you prefer not to use AWS tools, you can upload files directly in the web interface.
If you prefer not to use command-line tools, you can upload files directly in the web interface.

### How to upload via the UI

1. Go to your product page in Source Cooperative (e.g. `https://source.coop/your-org/your-product`)
2. In the top-right corner of the Product Contents card, click on the lock icon to open the dropdown menu and enable edit mode by clicking on `Edit Mode`.
3. Either:
- Drag‑and‑drop files to the `Product Contents` card
- Use `Upload Files` or `Upload Directory` options in dropdown menu to upload files via operating system's file selectory
- Use `Upload Files` or `Upload Directory` options in dropdown menu to upload files via operating system's file selector

Your files will be uploaded automatically to your product.

Expand All @@ -45,13 +75,14 @@ This option is ideal for:

---

## Option 2: Upload using temporary AWS credentials (recommended for larger uploads)
## Option 2: Upload through the data proxy with temporary credentials (recommended for larger uploads)

For larger uploads, scripting, or programmatic access, use temporary AWS credentials.
For larger uploads, scripting, or programmatic access, use temporary credentials
issued by Source Cooperative with an S3 client pointed at the data proxy.

### Get credentials with the Source CLI (recommended)

The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you and provides temporary AWS credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS profile, the AWS CLI and SDKs refresh credentials for you.
The [Source CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you with Source Cooperative and provides temporary credentials automatically, so you don't have to copy expiring credentials out of the UI. Once configured as an AWS CLI profile, the AWS CLI and SDKs refresh credentials for you, and you'll only need to re-authenticate every 30 days.

**1. Install the CLI**

Expand All @@ -65,17 +96,19 @@ source-coop login

This opens your browser to authenticate and caches short‑lived credentials in your OS keyring.

**3. Configure an AWS profile**
**3. Configure an AWS CLI profile**

Add a profile to `~/.aws/config` that calls the CLI to fetch credentials on demand:
Add a profile to `~/.aws/config` that calls the Source CLI to fetch credentials on demand and sends requests to the data proxy:

```ini
[profile source-coop]
credential_process = source-coop creds
endpoint_url = https://data.source.coop
```

**4. Use standard AWS commands with the profile**
The `endpoint_url` line is what sends requests to the Source data proxy instead of AWS.

**4. Use standard S3 commands with the profile**

Pass the profile per command with `--profile`:

Expand Down Expand Up @@ -112,12 +145,17 @@ The AWS CLI will refresh credentials automatically; re-run `source-coop login` w
You will also see:

- `Expiration`: the expiration time of the credentials (a specific date and time)
- `Bucket`: the bucket name (`your-org`)
- `Bucket`: the bucket name on the data proxy, which is your account ID (`your-org`)
- `Prefix`: the prefix (folder) you are allowed to write to (e.g. `your-product/`)

---

## What the credentials look ilke
## What the credentials look like

These are Source Cooperative credentials in the AWS credential format. They
only work against the data proxy, so always set the endpoint to
`https://data.source.coop`. The region value is required by S3 clients; it does
not say where your data is stored.

### For SDK clients (boto3, AWS SDKs, etc.)

Expand All @@ -130,6 +168,15 @@ You will also see:
}
```

With boto3, pass the endpoint when creating the client:

```python
import boto3

s3 = boto3.client("s3", endpoint_url="https://data.source.coop", **credentials)
s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv")
```

---

### For terminal / shell usage
Expand All @@ -139,6 +186,7 @@ export AWS_ACCESS_KEY_ID="ASIA..."
export AWS_SECRET_ACCESS_KEY="pwEV..."
export AWS_SESSION_TOKEN="IQoJ..."
export AWS_DEFAULT_REGION="us-west-2"
export AWS_ENDPOINT_URL="https://data.source.coop" # add this: send requests to the data proxy
```

These credentials are:
Expand All @@ -151,17 +199,13 @@ These credentials are:

## Where to upload your data

You may upload only to the provided prefix, for example:
You may upload only to your product's prefix on the data proxy, for example:

```
s3://us-west-2.opendata.source.coop/your-org/your-product/
s3://your-org/your-product/
```

or if you are not uploading under an organization:

```
s3://us-west-2.opendata.source.coop/your-product/
```
This matches your product page URL, `https://source.coop/your-org/your-product`.

You may upload:

Expand All @@ -176,15 +220,17 @@ You may not upload outside this path.
## Example: Upload using the AWS CLI

```bash
aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv
aws s3 cp mydata.csv s3://your-org/your-product/mydata.csv --endpoint-url https://data.source.coop
```

Or upload a full directory:

```bash
aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/
aws s3 sync ./data s3://your-org/your-product/ --endpoint-url https://data.source.coop
```

`--endpoint-url` isn't needed if you set `AWS_ENDPOINT_URL` or use the `source-coop` profile above.

---

## Option 3: Long‑standing or automated access (ADVANCED)
Expand All @@ -197,11 +243,21 @@ If you need:

You can use your own IAM role to write to the Source Cooperative bucket.

:::info This option really is AWS

Unlike Options 1 and 2, this option bypasses the data proxy. You write
directly to Source Cooperative's Amazon S3 bucket (e.g. `us-west-2.opendata.source.coop`, `eu-west-1.opendata.source.coop`)
using an identity in **your own AWS account**. It only applies to products stored
Comment thread
alukach marked this conversation as resolved.
in that bucket. If your product is stored with another provider, or you don't have
an AWS account, use [Option 2](#option-2-upload-through-the-data-proxy-with-temporary-credentials-recommended-for-larger-uploads)
or contact us.

:::

### How this works

- You create an IAM role (or use an existing IAM user/account) in **your own** AWS account
- You send us its ARN, and we grant it write access to your account's prefix in the Source Cooperative bucket
- You upload with `--acl bucket-owner-full-control` so Source Cooperative owns the objects

No credentials are shared, and no role chaining is required.

Expand Down Expand Up @@ -287,7 +343,7 @@ in-progress uploads across the *whole* bucket. AWS does not support the
`s3:prefix` condition key on it, so it cannot be limited to your data, and no
upload path needs it — incomplete uploads are cleaned up automatically after 7
days. Only `aws s3api list-multipart-uploads` requires it; contact us if you
have a workflow that does.
have a workflow that requires this policy.

:::

Expand Down Expand Up @@ -334,11 +390,10 @@ We will add the ARN to the bucket policy and confirm when it is active.

### Step 3: Upload using that identity

Once we confirm, upload with credentials for that role, user, or account. **You must set `bucket-owner-full-control`** so Source Cooperative fully owns and can manage the uploaded objects:
Once we confirm, upload with credentials for that role, user, or account:

```bash
aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv \
--acl bucket-owner-full-control
aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv
```

Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) need no assume-role step — the SDK picks up the role automatically.
Expand All @@ -347,8 +402,7 @@ Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles)
<summary>Uploading a directory</summary>

```bash
aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ \
--acl bucket-owner-full-control
aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/
```

</details>
Expand All @@ -367,7 +421,6 @@ region = us-west-2

```bash
aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ \
--acl bucket-owner-full-control \
--profile source-coop-upload
```

Expand All @@ -384,7 +437,6 @@ s3.upload_file(
"mydata.csv",
"us-west-2.opendata.source.coop",
"your-org/your-product/mydata.csv",
ExtraArgs={"ACL": "bucket-owner-full-control"},
)
```

Expand All @@ -408,11 +460,10 @@ This protects both you and Source Cooperative.

Please do not:

@tylere tylere Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a user tries any of these actions they will fail, correct?

@alukach alukach Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the actions that aren't human behavior, yes, they will be denied.


- Request permanent AWS access keys
- Request permanent access keys
- Ask for full bucket access
- Upload outside your assigned prefix
- Reuse expired temporary credentials
- Upload without `bucket-owner-full-control` when using your own role

---

Expand Down