feat: exchange a service account's API key without a browser

README.md

@@ -2,7 +2,7 @@
 
 Authenticate with the Source Cooperative data proxy and obtain temporary S3 credentials.
 
-Uses the OAuth2 Authorization Code flow with PKCE to authenticate via browser, then exchanges the OIDC ID token at the proxy's STS endpoint for temporary AWS credentials.
+Uses the OAuth2 Authorization Code flow with PKCE to authenticate via browser, then exchanges the OIDC ID token at the proxy's STS endpoint for temporary AWS credentials. Software that runs unattended can instead exchange a service account's API key, with no browser (see [Using an API key](#using-an-api-key-no-browser)).
 
 ## Install
 
@@ -57,7 +57,9 @@ endpoint_url = https://data.source.coop
 aws s3 ls s3://my-account/my-product --profile source-coop
 ```
 
-When credentials expire, `source-coop creds` uses the cached refresh token to fetch new ones automatically. Run `source-coop login` again only when that fails (e.g. the refresh token has expired or been revoked).
+When credentials are about to expire, `source-coop creds` uses the cached refresh token to fetch new ones automatically. Run `source-coop login` again only when that fails (e.g. the refresh token has expired or been revoked).
+
+`creds` replaces credentials once they have 16 minutes left, or half the session if that is shorter. AWS SDKs ask `credential_process` for new credentials 5 to 15 minutes before expiry, so they get fresh ones on the first ask instead of rerunning `creds` for every request. If replacing them fails, `creds` serves the cached credentials, with a warning, until they expire.
 
 ### Logging in on a remote server (no browser)
 
@@ -79,6 +81,77 @@ source-coop login --port 8400
 
 Credentials are cached on the server (see [File fallback](#file-fallback)), and `source-coop creds` refreshes them there without another browser login.
 
+### Using an API key (no browser)
+
+Software that runs unattended, such as a cron job, a daemon or an instrument, authenticates as a service account with an API key (`sck_…`) rather than a browser login. `creds` exchanges the key at the proxy's STS endpoint, caches the credentials as it does for `login`, and exchanges the key again before they expire, so a long-running process keeps working without a person.
+
+1. Save the key, which is shown only once when you issue it, to a file only you can read:
+
+```bash
+mkdir -p ~/.config/source-coop
+(umask 077 && cat > ~/.config/source-coop/key)   # paste the key, press Enter, then Ctrl-D
+```
+
+2. Point `credential_process` at it in `~/.aws/config`, using the full path (`~` is not expanded there):
+
+```ini
+[profile source-coop]
+credential_process = source-coop creds --api-key-file /home/me/.config/source-coop/key
+endpoint_url = https://data.source.coop
+```
+
+3. Use AWS tools as usual:
+
+```bash
+aws s3 ls s3://my-account/my-product --profile source-coop
+```
+
+Instead of `--api-key-file`, the environment can supply the key: `SOURCE_API_KEY_FILE` names the file, or `SOURCE_API_KEY` holds the key itself. The file wins if both are set. No flag takes the key itself, because other users on the machine can read a command line.
+
+| Flag | Env var | Default | Description |
+|------|---------|---------|-------------|
+| `--api-key-file` | `SOURCE_API_KEY_FILE` | | File holding the API key |
+| | `SOURCE_API_KEY` | | The API key itself |
+| `--role-arn` | `SOURCE_ROLE_ARN` | `_default` | Role to assume: a name such as `ReadOnly` (sent as `arn:aws:iam::000000000000:role/ReadOnly`) or a full ARN; see [Multiple roles](#multiple-roles) |
+| `--proxy-url` | `SOURCE_PROXY_URL` | `https://data.source.coop` | Proxy whose `/.sts` exchanges the key |
+| `--duration` | | | Session duration, e.g. `3600`, `90s`, `5m`, `12h`, `1d` |
+
+If the proxy refuses the key, `creds` prints the proxy's error and exits non-zero. Quote the request id when you contact support:
+
+```
+Error: STS error (InvalidIdentityToken): API key was not accepted (request id a40cee47fef1c4b4)
+```
+
+#### GDAL
+
+GDAL 3.12 and later run `credential_process` from the profile too. GDAL can't exchange the key on its own, because it sends its STS request as a GET with the token in the URL, and the proxy refuses a key in a URL. It gets credentials through the CLI instead. GDAL ignores the profile's `endpoint_url`, so name the proxy in `AWS_S3_ENDPOINT`:
+
+```bash
+AWS_PROFILE=source-coop AWS_S3_ENDPOINT=https://data.source.coop AWS_VIRTUAL_HOSTING=FALSE \
+  gdalinfo /vsis3/my-account/my-product/image.tif
+```
+
+With an older GDAL, export credentials into the environment instead. They are not refreshed, so run this again before they expire:
+
+```bash
+eval "$(source-coop creds --api-key-file ~/.config/source-coop/key --format env)"
+```
+
+#### Without the CLI
+
+AWS SDKs and the AWS CLI can exchange the key themselves and refresh on their own, with nothing else installed. Point `AWS_WEB_IDENTITY_TOKEN_FILE` at the key file and set four more variables:
+
+```bash
+export AWS_WEB_IDENTITY_TOKEN_FILE=$HOME/.config/source-coop/key
+export AWS_ROLE_ARN=arn:aws:iam::000000000000:role/_default
+export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts
+export AWS_ENDPOINT_URL_S3=https://data.source.coop
+export AWS_REGION=us-west-2   # required by the SDK; says nothing about where data lives
+aws s3 ls s3://my-account/my-product/
+```
+
+This needs an SDK that reads `AWS_ENDPOINT_URL_STS`: the AWS CLI 2.13 or later, boto3/botocore 1.31 or later, or a current Go v2, JavaScript v3 or Java 2.x SDK. `aws --debug` prints the key, so don't share its output.
+
 ### Checking the CLI version
 
 ```bash
@@ -100,7 +173,7 @@ This sets `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`
 > [!WARNING]
 > Custom roles are not yet supported within the Source Cooperative data proxy. 
 
-Each role's credentials are cached separately:
+A bare role name such as `reader-role` reaches the proxy as `arn:aws:iam::000000000000:role/reader-role`, the ARN form AWS SDKs send; a full ARN is sent as given. Each role's credentials are cached separately:
 
 ```bash
 source-coop login --role-arn reader-role
@@ -126,7 +199,7 @@ endpoint_url = https://data.source.coop
 | `--issuer` | `SOURCE_OIDC_ISSUER` | `https://auth.source.coop` | OIDC issuer URL |
 | `--client-id` | `SOURCE_OIDC_CLIENT_ID` | `d037d00b-...` | OAuth2 client ID |
 | `--proxy-url` | `SOURCE_PROXY_URL` | `https://data.source.coop` | S3 proxy URL for STS |
-| `--role-arn` | `SOURCE_ROLE_ARN` | `source-coop-user` | Role ARN to assume |
+| `--role-arn` | `SOURCE_ROLE_ARN` | `_default` | Role to assume: a name such as `ReadOnly`, or a full ARN |
 | `--format` | | `credential-process` | Output format: `credential-process`, `env`, or `aws-credentials` |
 | `--profile` | | `source-coop` | Profile name for `--format aws-credentials` |
 | `--duration` | | | Session duration, e.g. `3600`, `90s`, `5m`, `12h`, `1d` (bare number = seconds) |
@@ -175,7 +248,7 @@ The CLI caches temporary STS credentials so that `creds` can output them without
 
 ### OS keyring (default)
 
-Credentials are stored in the OS-native keyring under the service name `source-coop-cli`, keyed by role ARN:
+Credentials are stored in the OS-native keyring under the service name `source-coop-cli`, keyed by role. An API key's credentials are keyed by role and a prefix of the key's SHA-256, so they never mix with a `login` session's or another key's; the key itself is never stored.
 
 | Platform | Backend |
 |----------|---------|

src/cache.rs

@@ -1,6 +1,7 @@
 use crate::sts::Credentials;
 use chrono::Utc;
 use serde::{Deserialize, Serialize};
+use sha2::{Digest, Sha256};
 use std::fs;
 use std::io;
 use std::path::PathBuf;
@@ -65,6 +66,19 @@ fn cache_path(role_arn: &str) -> Result<PathBuf, String> {
         .join(format!("{sanitized}.json")))
 }
 
+/// The slot an API key's credentials are cached under: the role plus the
+/// first 16 hex digits of the key's SHA-256. `login` caches under the role
+/// alone, so a key's credentials never share a slot with a person's session
+/// or another key's, and the key itself is never written down.
+pub fn api_key_slot(key: &str, role_arn: &str) -> String {
+    let hash: String = Sha256::digest(key)
+        .iter()
+        .take(8)
+        .map(|b| format!("{b:02x}"))
+        .collect();
+    format!("{role_arn}+key-{hash}")
+}
+
 /// Take an exclusive per-role lock, held until the returned file is dropped.
 /// Refresh tokens rotate on use, and the IdP may revoke the whole token family
 /// if an old one is replayed, so concurrent `creds` calls must not refresh in
@@ -178,17 +192,30 @@ pub fn read_credentials(role_arn: &str) -> Result<Option<CacheEntry>, String> {
 
 /// Check if credentials are expired or will expire within a 60-second buffer.
 pub fn is_expired(creds: &Credentials) -> Result<bool, String> {
+    expires_within(creds, 60)
+}
+
+/// Whether to replace credentials from a session of `duration` seconds (the
+/// proxy's default hour when `None`) before handing them out. botocore
+/// refreshes credential_process credentials with under 15 minutes left, and
+/// reruns the process on every lookup until it gets ones with more, so they are
+/// replaced with 16 minutes left, the extra minute for clock skew. A shorter
+/// session is replaced halfway through instead, or every call would replace
+/// it, and never later than the one-minute buffer.
+pub fn needs_refresh(creds: &Credentials, duration: Option<u64>) -> Result<bool, String> {
+    let margin = (duration.unwrap_or(3600) / 2).clamp(60, 16 * 60);
+    expires_within(creds, margin as i64)
+}
+
+fn expires_within(creds: &Credentials, seconds: i64) -> Result<bool, String> {
     let expiration = chrono::DateTime::parse_from_rfc3339(&creds.expiration).map_err(|e| {
         format!(
             "Failed to parse expiration timestamp '{}': {e}",
             creds.expiration
         )
     })?;
 
-    let now = Utc::now();
-    let buffer = chrono::Duration::seconds(60);
-
-    Ok(expiration <= now + buffer)
+    Ok(expiration <= Utc::now() + chrono::Duration::seconds(seconds))
 }
 
 #[cfg(test)]
@@ -222,6 +249,15 @@ mod tests {
         assert_eq!(sanitize_role_arn("my_role-name"), "my_role-name");
     }
 
+    #[test]
+    fn api_key_slots_are_per_key_and_role_and_hide_the_key() {
+        let slot = api_key_slot("sck_a", "_default");
+        assert_ne!(slot, "_default", "must not be login's slot for the role");
+        assert_ne!(slot, api_key_slot("sck_b", "_default"));
+        assert_ne!(slot, api_key_slot("sck_a", "ReadOnly"));
+        assert!(!slot.contains("sck_a"));
+    }
+
     #[test]
     fn expired_future_date() {
         let future = (Utc::now() + chrono::Duration::hours(1)).to_rfc3339();
@@ -244,6 +280,17 @@ mod tests {
         assert!(is_expired(&creds).unwrap());
     }
 
+    #[test]
+    fn refresh_is_due_with_16_minutes_left_or_half_the_session() {
+        let left =
+            |minutes| sample_creds(&(Utc::now() + chrono::Duration::minutes(minutes)).to_rfc3339());
+        // Inside botocore's 15-minute window at the default hour.
+        assert!(needs_refresh(&left(10), None).unwrap());
+        // A 15-minute session isn't replaced until halfway through.
+        assert!(!needs_refresh(&left(10), Some(900)).unwrap());
+        assert!(!needs_refresh(&left(30), Some(3600)).unwrap());
+    }
+
     #[test]
     fn expired_invalid_timestamp() {
         let creds = sample_creds("not-a-timestamp");