diff --git a/README.md b/README.md index 2d0143c..6dcfd05 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Authenticate with the Source Cooperative data proxy and obtain temporary S3 credentials. -Uses the OAuth2 Authorization Code flow with PKCE to authenticate via browser, then exchanges the OIDC ID token at the proxy's STS endpoint for temporary AWS credentials. +Uses the OAuth2 Authorization Code flow with PKCE to authenticate via browser, then exchanges the OIDC ID token at the proxy's STS endpoint for temporary AWS credentials. Software that runs unattended can instead exchange a service account's API key, with no browser (see [Using an API key](#using-an-api-key-no-browser)). ## Install @@ -57,7 +57,9 @@ endpoint_url = https://data.source.coop aws s3 ls s3://my-account/my-product --profile source-coop ``` -When credentials expire, `source-coop creds` uses the cached refresh token to fetch new ones automatically. Run `source-coop login` again only when that fails (e.g. the refresh token has expired or been revoked). +When credentials are about to expire, `source-coop creds` uses the cached refresh token to fetch new ones automatically. Run `source-coop login` again only when that fails (e.g. the refresh token has expired or been revoked). + +`creds` replaces credentials once they have 16 minutes left, or half the session if that is shorter. AWS SDKs ask `credential_process` for new credentials 5 to 15 minutes before expiry, so they get fresh ones on the first ask instead of rerunning `creds` for every request. If replacing them fails, `creds` serves the cached credentials, with a warning, until they expire. ### Logging in on a remote server (no browser) @@ -79,6 +81,77 @@ source-coop login --port 8400 Credentials are cached on the server (see [File fallback](#file-fallback)), and `source-coop creds` refreshes them there without another browser login. +### Using an API key (no browser) + +Software that runs unattended, such as a cron job, a daemon or an instrument, authenticates as a service account with an API key (`sck_…`) rather than a browser login. `creds` exchanges the key at the proxy's STS endpoint, caches the credentials as it does for `login`, and exchanges the key again before they expire, so a long-running process keeps working without a person. + +1. Save the key, which is shown only once when you issue it, to a file only you can read: + +```bash +mkdir -p ~/.config/source-coop +(umask 077 && cat > ~/.config/source-coop/key) # paste the key, press Enter, then Ctrl-D +``` + +2. Point `credential_process` at it in `~/.aws/config`, using the full path (`~` is not expanded there): + +```ini +[profile source-coop] +credential_process = source-coop creds --api-key-file /home/me/.config/source-coop/key +endpoint_url = https://data.source.coop +``` + +3. Use AWS tools as usual: + +```bash +aws s3 ls s3://my-account/my-product --profile source-coop +``` + +Instead of `--api-key-file`, the environment can supply the key: `SOURCE_API_KEY_FILE` names the file, or `SOURCE_API_KEY` holds the key itself. The file wins if both are set. No flag takes the key itself, because other users on the machine can read a command line. + +| Flag | Env var | Default | Description | +|------|---------|---------|-------------| +| `--api-key-file` | `SOURCE_API_KEY_FILE` | | File holding the API key | +| | `SOURCE_API_KEY` | | The API key itself | +| `--role-arn` | `SOURCE_ROLE_ARN` | `_default` | Role to assume: a name such as `ReadOnly` (sent as `arn:aws:iam::000000000000:role/ReadOnly`) or a full ARN; see [Multiple roles](#multiple-roles) | +| `--proxy-url` | `SOURCE_PROXY_URL` | `https://data.source.coop` | Proxy whose `/.sts` exchanges the key | +| `--duration` | | | Session duration, e.g. `3600`, `90s`, `5m`, `12h`, `1d` | + +If the proxy refuses the key, `creds` prints the proxy's error and exits non-zero. Quote the request id when you contact support: + +``` +Error: STS error (InvalidIdentityToken): API key was not accepted (request id a40cee47fef1c4b4) +``` + +#### GDAL + +GDAL 3.12 and later run `credential_process` from the profile too. GDAL can't exchange the key on its own, because it sends its STS request as a GET with the token in the URL, and the proxy refuses a key in a URL. It gets credentials through the CLI instead. GDAL ignores the profile's `endpoint_url`, so name the proxy in `AWS_S3_ENDPOINT`: + +```bash +AWS_PROFILE=source-coop AWS_S3_ENDPOINT=https://data.source.coop AWS_VIRTUAL_HOSTING=FALSE \ + gdalinfo /vsis3/my-account/my-product/image.tif +``` + +With an older GDAL, export credentials into the environment instead. They are not refreshed, so run this again before they expire: + +```bash +eval "$(source-coop creds --api-key-file ~/.config/source-coop/key --format env)" +``` + +#### Without the CLI + +AWS SDKs and the AWS CLI can exchange the key themselves and refresh on their own, with nothing else installed. Point `AWS_WEB_IDENTITY_TOKEN_FILE` at the key file and set four more variables: + +```bash +export AWS_WEB_IDENTITY_TOKEN_FILE=$HOME/.config/source-coop/key +export AWS_ROLE_ARN=arn:aws:iam::000000000000:role/_default +export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts +export AWS_ENDPOINT_URL_S3=https://data.source.coop +export AWS_REGION=us-west-2 # required by the SDK; says nothing about where data lives +aws s3 ls s3://my-account/my-product/ +``` + +This needs an SDK that reads `AWS_ENDPOINT_URL_STS`: the AWS CLI 2.13 or later, boto3/botocore 1.31 or later, or a current Go v2, JavaScript v3 or Java 2.x SDK. `aws --debug` prints the key, so don't share its output. + ### Checking the CLI version ```bash @@ -100,7 +173,7 @@ This sets `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` > [!WARNING] > Custom roles are not yet supported within the Source Cooperative data proxy. -Each role's credentials are cached separately: +A bare role name such as `reader-role` reaches the proxy as `arn:aws:iam::000000000000:role/reader-role`, the ARN form AWS SDKs send; a full ARN is sent as given. Each role's credentials are cached separately: ```bash source-coop login --role-arn reader-role @@ -126,7 +199,7 @@ endpoint_url = https://data.source.coop | `--issuer` | `SOURCE_OIDC_ISSUER` | `https://auth.source.coop` | OIDC issuer URL | | `--client-id` | `SOURCE_OIDC_CLIENT_ID` | `d037d00b-...` | OAuth2 client ID | | `--proxy-url` | `SOURCE_PROXY_URL` | `https://data.source.coop` | S3 proxy URL for STS | -| `--role-arn` | `SOURCE_ROLE_ARN` | `source-coop-user` | Role ARN to assume | +| `--role-arn` | `SOURCE_ROLE_ARN` | `_default` | Role to assume: a name such as `ReadOnly`, or a full ARN | | `--format` | | `credential-process` | Output format: `credential-process`, `env`, or `aws-credentials` | | `--profile` | | `source-coop` | Profile name for `--format aws-credentials` | | `--duration` | | | Session duration, e.g. `3600`, `90s`, `5m`, `12h`, `1d` (bare number = seconds) | @@ -175,7 +248,7 @@ The CLI caches temporary STS credentials so that `creds` can output them without ### OS keyring (default) -Credentials are stored in the OS-native keyring under the service name `source-coop-cli`, keyed by role ARN: +Credentials are stored in the OS-native keyring under the service name `source-coop-cli`, keyed by role. An API key's credentials are keyed by role and a prefix of the key's SHA-256, so they never mix with a `login` session's or another key's; the key itself is never stored. | Platform | Backend | |----------|---------| diff --git a/src/cache.rs b/src/cache.rs index b7c5bad..2615dd9 100644 --- a/src/cache.rs +++ b/src/cache.rs @@ -1,6 +1,7 @@ use crate::sts::Credentials; use chrono::Utc; use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; use std::fs; use std::io; use std::path::PathBuf; @@ -65,6 +66,19 @@ fn cache_path(role_arn: &str) -> Result { .join(format!("{sanitized}.json"))) } +/// The slot an API key's credentials are cached under: the role plus the +/// first 16 hex digits of the key's SHA-256. `login` caches under the role +/// alone, so a key's credentials never share a slot with a person's session +/// or another key's, and the key itself is never written down. +pub fn api_key_slot(key: &str, role_arn: &str) -> String { + let hash: String = Sha256::digest(key) + .iter() + .take(8) + .map(|b| format!("{b:02x}")) + .collect(); + format!("{role_arn}+key-{hash}") +} + /// Take an exclusive per-role lock, held until the returned file is dropped. /// Refresh tokens rotate on use, and the IdP may revoke the whole token family /// if an old one is replayed, so concurrent `creds` calls must not refresh in @@ -178,6 +192,22 @@ pub fn read_credentials(role_arn: &str) -> Result, String> { /// Check if credentials are expired or will expire within a 60-second buffer. pub fn is_expired(creds: &Credentials) -> Result { + expires_within(creds, 60) +} + +/// Whether to replace credentials from a session of `duration` seconds (the +/// proxy's default hour when `None`) before handing them out. botocore +/// refreshes credential_process credentials with under 15 minutes left, and +/// reruns the process on every lookup until it gets ones with more, so they are +/// replaced with 16 minutes left, the extra minute for clock skew. A shorter +/// session is replaced halfway through instead, or every call would replace +/// it, and never later than the one-minute buffer. +pub fn needs_refresh(creds: &Credentials, duration: Option) -> Result { + let margin = (duration.unwrap_or(3600) / 2).clamp(60, 16 * 60); + expires_within(creds, margin as i64) +} + +fn expires_within(creds: &Credentials, seconds: i64) -> Result { let expiration = chrono::DateTime::parse_from_rfc3339(&creds.expiration).map_err(|e| { format!( "Failed to parse expiration timestamp '{}': {e}", @@ -185,10 +215,7 @@ pub fn is_expired(creds: &Credentials) -> Result { ) })?; - let now = Utc::now(); - let buffer = chrono::Duration::seconds(60); - - Ok(expiration <= now + buffer) + Ok(expiration <= Utc::now() + chrono::Duration::seconds(seconds)) } #[cfg(test)] @@ -222,6 +249,15 @@ mod tests { assert_eq!(sanitize_role_arn("my_role-name"), "my_role-name"); } + #[test] + fn api_key_slots_are_per_key_and_role_and_hide_the_key() { + let slot = api_key_slot("sck_a", "_default"); + assert_ne!(slot, "_default", "must not be login's slot for the role"); + assert_ne!(slot, api_key_slot("sck_b", "_default")); + assert_ne!(slot, api_key_slot("sck_a", "ReadOnly")); + assert!(!slot.contains("sck_a")); + } + #[test] fn expired_future_date() { let future = (Utc::now() + chrono::Duration::hours(1)).to_rfc3339(); @@ -244,6 +280,17 @@ mod tests { assert!(is_expired(&creds).unwrap()); } + #[test] + fn refresh_is_due_with_16_minutes_left_or_half_the_session() { + let left = + |minutes| sample_creds(&(Utc::now() + chrono::Duration::minutes(minutes)).to_rfc3339()); + // Inside botocore's 15-minute window at the default hour. + assert!(needs_refresh(&left(10), None).unwrap()); + // A 15-minute session isn't replaced until halfway through. + assert!(!needs_refresh(&left(10), Some(900)).unwrap()); + assert!(!needs_refresh(&left(30), Some(3600)).unwrap()); + } + #[test] fn expired_invalid_timestamp() { let creds = sample_creds("not-a-timestamp"); diff --git a/src/main.rs b/src/main.rs index 6516a00..b76b469 100644 --- a/src/main.rs +++ b/src/main.rs @@ -4,6 +4,8 @@ mod output; mod sts; use clap::{Parser, Subcommand, ValueEnum}; +use std::fs; +use std::path::{Path, PathBuf}; #[cfg(feature = "staging")] mod defaults { @@ -56,8 +58,9 @@ struct Cli { enum Commands { /// Authenticate via OIDC and obtain temporary S3 credentials Login(LoginArgs), - /// Output cached credentials as credential_process JSON, shell env vars, - /// or an AWS credentials-file profile, refreshing them first if expired + /// Output credentials as credential_process JSON, shell env vars, or an + /// AWS credentials-file profile: those cached by `login` (refreshed first + /// if expired), or ones exchanged for a service account's API key Creds(CredsArgs), } @@ -75,7 +78,7 @@ struct LoginArgs { #[arg(long, env = "SOURCE_PROXY_URL", default_value = defaults::PROXY_URL)] proxy_url: String, - /// Role ARN to assume + /// Role to assume: a role name such as `ReadOnly`, or a full role ARN #[arg(long, env = "SOURCE_ROLE_ARN", default_value = defaults::ROLE_ARN)] role_arn: String, @@ -107,10 +110,26 @@ struct LoginArgs { #[derive(Parser)] struct CredsArgs { - /// Role ARN to read cached credentials for + /// Role to read cached credentials for, or with an API key to assume: a + /// role name such as `ReadOnly`, or a full role ARN #[arg(long, env = "SOURCE_ROLE_ARN", default_value = defaults::ROLE_ARN)] role_arn: String, + /// File holding a service account's API key (`sck_…`), exchanged for + /// credentials without a browser. SOURCE_API_KEY may hold the key itself + /// instead; this file wins if both are set + #[arg(long, env = "SOURCE_API_KEY_FILE")] + api_key_file: Option, + + /// S3 proxy URL for an API key's STS exchange (`login` records its own) + #[arg(long, env = "SOURCE_PROXY_URL", default_value = defaults::PROXY_URL)] + proxy_url: String, + + /// Session duration for an API key's exchange, e.g. `3600`, `90s`, `5m`, + /// `12h`, `1d` (bare number = seconds) + #[arg(long, value_parser = parse_duration)] + duration: Option, + /// Output format #[arg(long, default_value = "credential-process")] format: OutputFormat, @@ -210,10 +229,149 @@ async fn run_login(args: LoginArgs, verbose: bool) -> Result<(), String> { } async fn run_creds(args: CredsArgs, verbose: bool) -> Result<(), String> { + let api_key = api_key( + args.api_key_file.as_deref(), + std::env::var("SOURCE_API_KEY").ok(), + )?; + let creds = match api_key { + Some(key) => { + let slot = cache::api_key_slot(&key, &args.role_arn); + // One exchange at a time per key and role: callers started together + // (SDK threads, a batch of jobs) then read what the first one cached + // instead of each spending an exchange against the proxy's + // per-address rate limit. + let _lock = cache::lock(&slot).ok(); + let load = || cache::read_credentials(&slot); + let save = |e: &cache::CacheEntry| cache::write_credentials(&slot, e).map(drop); + key_credentials(&args, &key, verbose, load, save).await? + } + None => login_credentials(&args, verbose).await?, + }; + + match args.format { + OutputFormat::CredentialProcess => output::print_credential_process(&creds), + OutputFormat::Env => output::print_env(&creds), + OutputFormat::AwsCredentials => output::print_aws_credentials(&creds, &args.profile), + } + Ok(()) +} + +/// The API key to exchange, if one is configured: the contents of +/// `--api-key-file`, which wins, else `env` (SOURCE_API_KEY). No flag takes the +/// key itself, because other users on the machine can read a command line. +fn api_key(file: Option<&Path>, env: Option) -> Result, String> { + let (raw, source) = match (file, env) { + (Some(path), _) => { + let raw = fs::read_to_string(path) + .map_err(|e| format!("Failed to read API key file {}: {e}", path.display()))?; + (raw, path.display().to_string()) + } + (None, Some(key)) => (key, "SOURCE_API_KEY".to_string()), + (None, None) => return Ok(None), + }; + let key = parse_api_key(&raw).ok_or_else(|| { + format!( + "{source} does not hold a Source API key (sck_ and 36 letters and digits); \ + check that it was copied whole" + ) + })?; + Ok(Some(key.to_string())) +} + +/// The key, trimmed because a key file ends in a newline, if it has an API +/// key's fixed shape and its checksum holds: `sck_`, 30 random base62 +/// characters, and six more that are their checksum. Anything else, such as a +/// JWT, the wrong file's contents or a key cut short, is never sent as a key. +fn parse_api_key(raw: &str) -> Option<&str> { + let key = raw.trim(); + let body = key.strip_prefix("sck_")?.as_bytes(); + let is_key = body.len() == 36 + && body.iter().all(u8::is_ascii_alphanumeric) + && body[30..] == key_checksum(&body[..30]); + is_key.then_some(key) +} + +/// A key's last six characters: the CRC-32 of the thirty before them (IEEE, +/// as zlib computes it), in base62, most significant digit first. +fn key_checksum(body: &[u8]) -> [u8; 6] { + const BASE62: &[u8; 62] = b"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"; + let mut crc = !0u32; + for &b in body { + crc ^= u32::from(b); + for _ in 0..8 { + crc = if crc & 1 == 1 { + (crc >> 1) ^ 0xEDB8_8320 + } else { + crc >> 1 + }; + } + } + let mut n = !crc; + let mut digits = [b'0'; 6]; + for digit in digits.iter_mut().rev() { + *digit = BASE62[(n % 62) as usize]; + n /= 62; + } + digits +} + +/// Credentials for a service account's API key: the cached ones until they are +/// due for refresh, otherwise a new exchange, cached for the next call. The +/// cache only saves exchanges, so one that can't be read or written costs an +/// exchange instead of failing the call: a daemon holding a key keeps getting +/// credentials. +async fn key_credentials( + args: &CredsArgs, + key: &str, + verbose: bool, + load: impl FnOnce() -> Result, String>, + save: impl FnOnce(&cache::CacheEntry) -> Result<(), String>, +) -> Result { + let cached = load().ok().flatten().map(|entry| entry.creds); + if let Some(creds) = cached + .as_ref() + .filter(|c| cache::needs_refresh(c, args.duration) == Ok(false)) + { + return Ok(creds.clone()); + } + let exchanged = + sts::assume_role(&args.proxy_url, &args.role_arn, key, args.duration, verbose).await; + let entry = cache::CacheEntry { + creds: match exchanged { + Ok(creds) => creds, + Err(e) => return cached.and_then(|c| until_expiry(c, &e)).ok_or(e), + }, + refresh: None, + }; + if let Err(e) = save(&entry) { + eprintln!("Warning: credentials were not cached: {e}"); + } + Ok(entry.creds) +} + +/// The cached credentials after refreshing them early failed with `error`, if +/// they are still valid. An SDK keeps its credentials when an advisory refresh +/// fails, and so does this: refreshing early never ends a session sooner. +fn until_expiry(cached: sts::Credentials, error: &str) -> Option { + if cache::is_expired(&cached) != Ok(false) { + return None; + } + eprintln!("Warning: {error}; using cached credentials until they expire"); + Some(cached) +} + +/// `login`'s cached credentials for the role, refreshed first when due. +async fn login_credentials(args: &CredsArgs, verbose: bool) -> Result { const NOT_FOUND: &str = "No cached credentials found. Run 'source-coop login' first."; + // Only a refresh token replaces credentials without a person, so without + // one they are served until they expire rather than refreshed early. + let due = |e: &cache::CacheEntry| match &e.refresh { + Some(state) => cache::needs_refresh(&e.creds, state.duration), + None => cache::is_expired(&e.creds), + }; let mut entry = cache::read_credentials(&args.role_arn)?.ok_or(NOT_FOUND)?; - if cache::is_expired(&entry.creds)? { + if due(&entry)? { if entry.refresh.is_none() { return Err( "Cached credentials have expired. Run 'source-coop login' to refresh.".to_string(), @@ -222,21 +380,21 @@ async fn run_creds(args: CredsArgs, verbose: bool) -> Result<(), String> { // Re-read under the lock: another process may have refreshed already. let _lock = cache::lock(&args.role_arn)?; entry = cache::read_credentials(&args.role_arn)?.ok_or(NOT_FOUND)?; - if cache::is_expired(&entry.creds)? { + if due(&entry)? { + let cached = entry.creds.clone(); let save = |e: &cache::CacheEntry| cache::write_credentials(&args.role_arn, e).map(drop); - entry = refresh(&args.role_arn, entry, verbose, save).await.map_err(|e| { - format!("Cached credentials have expired and refresh failed ({e}). Run 'source-coop login'.") - })?; + entry = match refresh(&args.role_arn, entry, verbose, save).await { + Ok(entry) => entry, + Err(e) => { + return until_expiry(cached, &format!("refresh failed ({e})")).ok_or_else(|| { + format!("Cached credentials have expired and refresh failed ({e}). Run 'source-coop login'.") + }) + } + }; } } - - match args.format { - OutputFormat::CredentialProcess => output::print_credential_process(&entry.creds), - OutputFormat::Env => output::print_env(&entry.creds), - OutputFormat::AwsCredentials => output::print_aws_credentials(&entry.creds, &args.profile), - } - Ok(()) + Ok(entry.creds) } /// Trade the cached refresh token for a new id_token, exchange that for new STS @@ -276,7 +434,8 @@ async fn refresh( #[cfg(test)] mod tests { use super::*; - use wiremock::matchers::{body_string_contains, method, path, query_param}; + use std::collections::HashMap; + use wiremock::matchers::{body_string_contains, header, method, path, query_param_is_missing}; use wiremock::{Mock, MockServer, ResponseTemplate}; const STS_OK: &str = "\ @@ -316,12 +475,14 @@ mod tests { .await; } - /// Mock the proxy's STS endpoint: only the freshly issued id_token matches. + /// Mock the proxy's STS endpoint: only the freshly issued id_token, sent in + /// the form body rather than the URL, matches. async fn mock_sts(server: &MockServer, status: u16, body: &str) { - Mock::given(method("GET")) + Mock::given(method("POST")) .and(path("/.sts")) - .and(query_param("WebIdentityToken", "new-id-token")) - .and(query_param("DurationSeconds", "3600")) + .and(query_param_is_missing("WebIdentityToken")) + .and(body_string_contains("WebIdentityToken=new-id-token")) + .and(body_string_contains("DurationSeconds=3600")) .respond_with(ResponseTemplate::new(status).set_body_string(body)) .expect(1) .mount(server) @@ -415,6 +576,174 @@ mod tests { assert!(saved_entries.is_empty()); } + /// A throwaway key of the issued shape, built at run time so that secret + /// scanners looking for `sck_` keys don't flag this file. Its checksum was + /// computed independently, with Python's zlib.crc32. + fn test_key() -> String { + format!("sck_{}1EpiOw", "k".repeat(30)) + } + + fn in_minutes(minutes: i64) -> String { + (chrono::Utc::now() + chrono::Duration::minutes(minutes)).to_rfc3339() + } + + fn cached(access_key_id: &str, expiration: &str) -> cache::CacheEntry { + cache::CacheEntry { + creds: sts::Credentials { + access_key_id: access_key_id.into(), + secret_access_key: "SECRET".into(), + session_token: "SESSION".into(), + expiration: expiration.into(), + }, + refresh: None, + } + } + + /// Mock the proxy's STS endpoint for an API key: a form-encoded POST with + /// exactly these parameters and nothing in the URL, the only way the proxy + /// accepts a key. + async fn mock_key_sts(server: &MockServer, expected_calls: u64) { + let expected: HashMap = HashMap::from([ + ("Action".into(), "AssumeRoleWithWebIdentity".into()), + ( + "RoleArn".into(), + "arn:aws:iam::000000000000:role/ReadOnly".into(), + ), + ("WebIdentityToken".into(), test_key()), + ("DurationSeconds".into(), "900".into()), + ]); + Mock::given(method("POST")) + .and(path("/.sts")) + .and(header("content-type", "application/x-www-form-urlencoded")) + .and(move |req: &wiremock::Request| { + let form: HashMap = url::form_urlencoded::parse(&req.body) + .into_owned() + .collect(); + req.url.query().is_none() && form == expected + }) + .respond_with(ResponseTemplate::new(200).set_body_string(STS_OK)) + .expect(expected_calls) + .mount(server) + .await; + } + + /// Run `key_credentials` for the bare role `ReadOnly` against the mock + /// proxy, with `load` and `save` answering as given; returns the result and + /// the access key of each entry saved. + async fn run_key( + server: &MockServer, + load: Result, String>, + save: Result<(), String>, + ) -> (Result, Vec) { + let uri = server.uri(); + let args = CredsArgs::parse_from([ + "creds", + "--role-arn", + "ReadOnly", + "--proxy-url", + &uri, + "--duration", + "15m", + ]); + let mut saved = vec![]; + let result = key_credentials( + &args, + &test_key(), + false, + || load, + |e| { + saved.push(e.creds.access_key_id.clone()); + save + }, + ) + .await; + (result, saved) + } + + #[tokio::test] + async fn api_key_is_exchanged_again_before_credentials_expire() { + let server = MockServer::start().await; + mock_key_sts(&server, 1).await; + + // Five minutes left of the 15-minute session `run_key` asks for. + let due = cached("OLDKEY", &in_minutes(5)); + let (result, saved) = run_key(&server, Ok(Some(due)), Ok(())).await; + + assert_eq!(result.unwrap().access_key_id, "NEWKEY"); + assert_eq!(saved, ["NEWKEY"]); + } + + #[tokio::test] + async fn failed_early_exchange_serves_cached_credentials_until_they_expire() { + let server = MockServer::start().await; // answers every request 404 + + let due = cached("OLDKEY", &in_minutes(5)); + let (result, _) = run_key(&server, Ok(Some(due)), Ok(())).await; + assert_eq!(result.unwrap().access_key_id, "OLDKEY"); + + let expired = cached("OLDKEY", "2020-01-01T00:00:00Z"); + let (result, _) = run_key(&server, Ok(Some(expired)), Ok(())).await; + assert!(result.unwrap_err().contains("HTTP 404")); + } + + #[tokio::test] + async fn api_key_credentials_come_from_the_cache_while_fresh() { + let server = MockServer::start().await; + mock_key_sts(&server, 0).await; + + let fresh = cached("CACHEDKEY", "2099-01-01T00:00:00Z"); + let (result, saved) = run_key(&server, Ok(Some(fresh)), Ok(())).await; + + assert_eq!(result.unwrap().access_key_id, "CACHEDKEY"); + assert!(saved.is_empty()); + } + + #[tokio::test] + async fn api_key_is_exchanged_even_when_the_cache_is_unusable() { + let server = MockServer::start().await; + mock_key_sts(&server, 1).await; + + let locked = "keychain is locked".to_string(); + let (result, _) = run_key(&server, Err(locked.clone()), Err(locked)).await; + + assert_eq!(result.unwrap().access_key_id, "NEWKEY"); + } + + #[test] + fn only_a_key_shaped_token_is_sent_as_a_key() { + let key = test_key(); + assert_eq!(parse_api_key(&format!("{key}\r\n")), Some(key.as_str())); + for bad in [ + String::new(), + "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJ4In0.c2ln".to_string(), + key[..39].to_string(), + format!("{key}k"), + key.replace("sck_", "SCK_"), + format!("sck_j{}", &key[5..]), + format!("sck_{}!{}", "k".repeat(29), &key[34..]), + format!("sck_{}", "k".repeat(43)), + ] { + assert_eq!(parse_api_key(&bad), None, "{bad:?}"); + } + } + + #[test] + fn api_key_file_wins_over_env_and_errors_never_echo_a_key() { + let (key, other) = (test_key(), format!("sck_{}18XLyL", "o".repeat(30))); + let file = std::env::temp_dir().join(format!("source-coop-key-{}", std::process::id())); + fs::write(&file, format!("{key}\n")).unwrap(); + assert_eq!(api_key(Some(&file), Some(other.clone())), Ok(Some(key))); + fs::remove_file(&file).unwrap(); + + assert_eq!(api_key(None, Some(other.clone())), Ok(Some(other))); + assert_eq!(api_key(None, None), Ok(None)); + let err = api_key(None, Some("sck_not-quite".into())).unwrap_err(); + assert!( + err.starts_with("SOURCE_API_KEY") && !err.contains("not-quite"), + "{err}" + ); + } + #[test] fn parses_units_and_bare_seconds() { assert_eq!(parse_duration("3600").unwrap(), 3600); diff --git a/src/output.rs b/src/output.rs index 1863d05..8974ddb 100644 --- a/src/output.rs +++ b/src/output.rs @@ -1,7 +1,7 @@ use crate::sts::Credentials; -/// Print credentials in AWS credential_process JSON format. -pub fn print_credential_process(creds: &Credentials) { +/// Credentials in AWS credential_process JSON format. +fn credential_process_json(creds: &Credentials) -> String { let json = serde_json::json!({ "Version": 1, "AccessKeyId": creds.access_key_id, @@ -9,7 +9,12 @@ pub fn print_credential_process(creds: &Credentials) { "SessionToken": creds.session_token, "Expiration": creds.expiration, }); - println!("{}", serde_json::to_string_pretty(&json).unwrap()); + serde_json::to_string_pretty(&json).unwrap() +} + +/// Print credentials in AWS credential_process JSON format. +pub fn print_credential_process(creds: &Credentials) { + println!("{}", credential_process_json(creds)); } /// Print credentials as shell export statements. @@ -28,3 +33,30 @@ pub fn print_aws_credentials(creds: &Credentials, profile: &str) { println!("aws_secret_access_key = {}", creds.secret_access_key); println!("aws_session_token = {}", creds.session_token); } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn credential_process_json_has_the_fields_sdks_and_gdal_read() { + let creds = Credentials { + access_key_id: "AKID".into(), + secret_access_key: "SECRET".into(), + session_token: "TOKEN".into(), + expiration: "2099-01-01T00:00:00Z".into(), + }; + let json: serde_json::Value = + serde_json::from_str(&credential_process_json(&creds)).unwrap(); + assert_eq!( + json, + serde_json::json!({ + "Version": 1, + "AccessKeyId": "AKID", + "SecretAccessKey": "SECRET", + "SessionToken": "TOKEN", + "Expiration": "2099-01-01T00:00:00Z", + }) + ); + } +} diff --git a/src/sts.rs b/src/sts.rs index b7efd7d..1de97fa 100644 --- a/src/sts.rs +++ b/src/sts.rs @@ -18,47 +18,58 @@ fn sts_url(proxy_url: &str) -> Result { Ok(url) } -/// Call the proxy's STS AssumeRoleWithWebIdentity endpoint. +/// A role as the ARN a stock SDK sends in `AWS_ROLE_ARN`: a bare name such as +/// `ReadOnly` becomes `arn:aws:iam::000000000000:role/ReadOnly`. The proxy +/// ignores the account segment, because the token names the account; an ARN +/// passes through unchanged. +fn role_arn(role: &str) -> String { + if role.starts_with("arn:") { + role.to_string() + } else { + format!("arn:aws:iam::000000000000:role/{role}") + } +} + +/// Call the proxy's STS AssumeRoleWithWebIdentity endpoint. The token travels +/// in a form-encoded POST body, never the URL: URLs end up in access logs, and +/// the proxy refuses an API key sent in one. pub async fn assume_role( proxy_url: &str, - role_arn: &str, + role: &str, web_identity_token: &str, duration_seconds: Option, verbose: bool, ) -> Result { - let mut url = sts_url(proxy_url)?; - - url.query_pairs_mut() - .append_pair("Action", "AssumeRoleWithWebIdentity") - .append_pair("RoleArn", role_arn) - .append_pair("WebIdentityToken", web_identity_token); - - if let Some(duration) = duration_seconds { - url.query_pairs_mut() - .append_pair("DurationSeconds", &duration.to_string()); + let url = sts_url(proxy_url)?; + let role_arn = role_arn(role); + let duration = duration_seconds.map(|d| d.to_string()); + let mut form = vec![ + ("Action", "AssumeRoleWithWebIdentity"), + ("RoleArn", role_arn.as_str()), + ("WebIdentityToken", web_identity_token), + ]; + if let Some(duration) = &duration { + form.push(("DurationSeconds", duration)); } if verbose { - // Log the URL without the WebIdentityToken to avoid leaking secrets - let mut redacted_url = sts_url(proxy_url)?; - redacted_url - .query_pairs_mut() - .append_pair("Action", "AssumeRoleWithWebIdentity") - .append_pair("RoleArn", role_arn) - .append_pair("WebIdentityToken", ""); - if let Some(duration) = duration_seconds { - redacted_url - .query_pairs_mut() - .append_pair("DurationSeconds", &duration.to_string()); - } - eprintln!("[verbose] GET {redacted_url}"); + eprintln!("[verbose] POST {url}"); + eprintln!("[verbose] RoleArn={role_arn}"); } - let resp = reqwest::get(url.as_str()) + let resp = reqwest::Client::new() + .post(url) + .form(&form) + .send() .await .map_err(|e| format!("STS request failed: {e}"))?; let status = resp.status(); + let request_id = resp + .headers() + .get("x-request-id") + .and_then(|v| v.to_str().ok()) + .map(String::from); if verbose { eprintln!("[verbose] Response: {status}"); @@ -73,14 +84,16 @@ pub async fn assume_role( if verbose { eprintln!("[verbose] Response body:\n{body}"); } - // Try to parse error XML for a better message - if let Ok(err) = xml_from_str::(&body) { - return Err(format!( - "STS error ({}): {}", - err.error.code, err.error.message - )); + let mut error = match xml_from_str::(&body) { + Ok(err) => format!("STS error ({}): {}", err.error.code, err.error.message), + Err(_) => format!("STS request failed (HTTP {status}): {body}"), + }; + // Support finds the proxy's log lines by request id. A refused API key + // carries it in the message; other failures carry it only in a header. + if let Some(id) = request_id.filter(|id| !error.contains(id.as_str())) { + error.push_str(&format!(" (request id {id})")); } - return Err(format!("STS request failed (HTTP {status}): {body}")); + return Err(error); } let parsed: StsResponse = @@ -146,7 +159,9 @@ struct StsError { #[cfg(test)] mod tests { - use super::sts_url; + use super::*; + use wiremock::matchers::{method, path}; + use wiremock::{Mock, MockServer, ResponseTemplate}; #[test] fn targets_sts_path_from_base() { @@ -171,4 +186,56 @@ mod tests { assert_eq!(sts_url(input).unwrap().path(), "/.sts", "input: {input}"); } } + + #[test] + fn expands_bare_role_names_to_arns() { + let default = "arn:aws:iam::000000000000:role/_default"; + assert_eq!(role_arn("_default"), default); + assert_eq!(role_arn(default), default); + assert_eq!( + role_arn("ReadOnly"), + "arn:aws:iam::000000000000:role/ReadOnly" + ); + } + + /// The error `assume_role` returns when the proxy answers `status` with + /// `body`, tagged with a request id the way the proxy tags every response. + async fn sts_error(status: u16, body: &str) -> String { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/.sts")) + .respond_with( + ResponseTemplate::new(status) + .insert_header("x-request-id", "a40cee47fef1c4b4") + .set_body_string(body), + ) + .mount(&server) + .await; + assume_role(&server.uri(), "_default", "sck_x", None, false) + .await + .unwrap_err() + } + + #[tokio::test] + async fn surfaces_the_proxy_error_verbatim() { + // The proxy's refusal of an API key, byte for byte. + let body = "\n\ + InvalidIdentityToken\ + API key was not accepted (request id a40cee47fef1c4b4)\ + "; + assert_eq!( + sts_error(400, body).await, + "STS error (InvalidIdentityToken): API key was not accepted (request id a40cee47fef1c4b4)" + ); + } + + #[tokio::test] + async fn adds_the_request_id_when_the_message_lacks_it() { + let body = "InternalError\ + internal error"; + assert_eq!( + sts_error(500, body).await, + "STS error (InternalError): internal error (request id a40cee47fef1c4b4)" + ); + } }