Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .gitleaksignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,41 @@
4f94b88909f5c8951027b11bd862e91edb8bd1b1:DEPLOYMENT.md:curl-auth-user:232
4f94b88909f5c8951027b11bd862e91edb8bd1b1:DEPLOYMENT.md:curl-auth-user:317
4f94b88909f5c8951027b11bd862e91edb8bd1b1:DEPLOYMENT.md:curl-auth-user:320

# --- Issue #63: tag-only commits from the pre-history-scrub v0.2.0 /
# v0.2.0-beta tags. actions/checkout@v4's fetch-depth:0 force-updates tags
# (+refs/tags/*), so CI scans these abandoned tag tips, not main. Each of
# the 5 duplicate entries below is byte-identical content to an
# already-ignored main-ancestor commit above (same author/date/diff), just
# under the pre-rewrite hash. Each of the 7 was individually verified live
# or dead before being added here -- see SECRET_SCAN_CI_INVESTIGATION.md
# and SECRET_SCAN_RESOLUTION.md for the full trail.

# github-fine-grained-pat, .env:15 -- ROTATED/REVOKED. Verified: confirmed
# absent from the current GitHub token list entirely (only one unrelated
# fine-grained PAT exists today, different purpose, different creation
# date).
979ba0c0cb4ced0e233e38ef18436e2c54db71c6:.env:github-fine-grained-pat:15

# generic-api-key (RAGBIO_API_KEY), .env:8 -- ROTATED. Verified: direct
# comparison of the leaked value against the current live value confirms
# they differ.
cced81dacacf28eed07a4426fe4b116295573507:.env:generic-api-key:8

# jwt, docker-compose.yml:232 -- dead. Verified: JWT payload decoded from
# the leaked token, exp claim (~2026-05-25) has passed. Duplicate content
# of already-ignored 8e35ae534a8 above.
981226817e716889fbbdcd213f7010af17e6d47b:docker-compose.yml:jwt:232

# jwt, docker-compose.release.yml:213 -- dead. Verified: JWT payload
# decoded from the leaked token, exp claim (~2026-05-25) has passed.
# Duplicate content of already-ignored 9a12a77d00b above.
11c868fb5436a7aea541120ab4f64962c36706b8:docker-compose.release.yml:jwt:213

# curl-auth-user (Grafana/Prometheus basic-auth), DEPLOYMENT.md:232,317,320
# -- different/safe. Verified: direct comparison confirms the current
# GF_ADMIN_PASSWORD is not the leaked "omnibioai" default. Duplicate
# content of already-ignored 4f94b88909f above.
975460f350e777340dc2d83911f9c565f18c12c8:DEPLOYMENT.md:curl-auth-user:232
975460f350e777340dc2d83911f9c565f18c12c8:DEPLOYMENT.md:curl-auth-user:317
975460f350e777340dc2d83911f9c565f18c12c8:DEPLOYMENT.md:curl-auth-user:320
116 changes: 116 additions & 0 deletions SECRET_SCAN_CI_INVESTIGATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# Secret-Scan CI Investigation

Status: **Part 1 resolved. Part 2 blocked — needs your decision before any
`.gitleaksignore` change is made. Part 3 not started. Part 4 done separately
(see bottom).**

## Part 1 — Commit-count discrepancy: RESOLVED

Reproduced CI's exact result locally: `450 commits scanned` / `leaks found: 7`
(count differs slightly from CI's 437 only because my dev clone still carries
a few extra local-only branches beyond what a fresh CI clone has — see below;
this has no effect on which leaks are found, confirmed).

**Root cause, with evidence:**

1. `scripts/secret-scan.sh` doesn't take a ref — it mounts the whole `.git`
directory and runs `gitleaks git /repo`. Gitleaks' git-mode walks **every
ref in the repo** (branches *and* tags), not just the checked-out branch.
2. CI's `actions/checkout@v4` with `fetch-depth: 0` fetches with
`+refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*` — confirmed
from the actual job log (`gh api .../jobs/99201926572/logs`). The leading
`+` **force-updates every tag**, even a conflicting one.
3. My day-to-day local clone had **stale copies of the `v0.2.0` and
`v0.2.0-beta` tags**. Running the identical fetch CI uses produced:
`t [tag update] v0.2.0 -> v0.2.0` and `v0.2.0-beta -> v0.2.0-beta` — i.e.
my local tags were pointing somewhere else and got force-moved. A plain
`git fetch`/`git pull` never force-moves a conflicting tag, so an ordinary
dev clone silently keeps drifting from what CI actually scans.
4. Once local tags matched origin exactly, the local scan reproduced CI's
`leaks found: 7` immediately.
5. All 7 findings' commits are **unreachable from any branch** (`git branch
--all --contains` = empty for all seven) but **are reachable from the
`v0.2.0` / `v0.2.0-beta` tags** (`git tag --contains` confirms). They are
not part of `main`'s real history.
6. For 5 of the 7, I diffed the tag-only commit against the commit already
named in `.gitleaksignore` for the same file/rule/line — same author date,
same commit message, byte-identical `--stat` (same files, same
insertions/deletions) in every pair checked:
- `981226817e7...` (tag-only) ≡ `8e35ae534a8...` (already in
`.gitleaksignore`, ancestor of `main`) — both "feat: integration tests
- 121 passing, 0 failures", identical 9-file/1670-insertion diff.
- `11c868fb543...` (tag-only) ≡ `9a12a77d00b...` (already ignored) —
identical diff (License gate + UI changes).
- `975460f350e...` (tag-only, 3 findings) ≡ `4f94b88909f...` (already
ignored) — identical diff (Grafana/Prometheus datasource + Workbench
UI changes).

Conclusion: at some point `main`'s history was rewritten (commit hashes
changed for identical content — the parent chain differs, so every
descendant hash changed too), which is exactly why `main`'s current
history is clean (my very first local scan, before touching tags, found
**0 leaks in 389 commits**). The `v0.2.0`/`v0.2.0-beta` tags were never
moved forward and still point at the pre-rewrite snapshot, so they
permanently retain the old secret-containing blobs. This is *not* related
to the `backup/main-before-pr51-rebase` branch (checked — unrelated, much
later, Aug 16) — it's a separate, earlier rewrite.

**Net effect:** CI is not scanning `main`. It's (unintentionally) scanning
two abandoned release tags that predate a history-scrub, every single run.

## Part 2 — Triage of the 7 findings: 5 of 7 have a provenance answer, liveness is NOT verified for any

| # | File | Rule | Tag-only commit | Status |
|---|------|------|------|--------|
| 1 | `.env` | `github-fine-grained-pat` | `979ba0c0c` (2026-05-28) | **New** — no prior `.gitleaksignore` entry |
| 2 | `docker-compose.yml:232` | `jwt` | `981226817` (2026-05-25) | Duplicate of already-ignored `8e35ae5` (see above) |
| 3-5 | `DEPLOYMENT.md:232,317,320` | `curl-auth-user` | `975460f35` (2026-05-22) | Duplicate of already-ignored `4f94b8890` (Grafana/Prometheus basic-auth) |
| 6 | `.env` (`RAGBIO_API_KEY`) | `generic-api-key` | `cced81dac` (2026-05-22) | **New** — no prior `.gitleaksignore` entry |
| 7 | `docker-compose.release.yml:213` | `jwt` | `11c868fb5` (2026-05-27) | Duplicate of already-ignored `9a12a77d0` |

Structural checks that came back clean:
- Current `main` no longer hardcodes any of these — `docker-compose.yml`'s
`JWT_SECRET`/`AUTH_SECRET_KEY` is `${AUTH_SECRET_KEY:?...}`, and `.env` is
untracked + gitignored on `main`.
- The existing `.gitleaksignore` comment ("Reviewed historical findings...")
is a **blanket** justification, not five individual ones — it doesn't say
*why* each is safe (rotated? fixture? placeholder?), so I can't treat it as
the individual verification Part 2 requires even for the 5 duplicates.

**What I could not verify, and why I stopped here rather than guess:**
Whether any of these five values — a GitHub fine-grained PAT, a
`RAGBIO_API_KEY`, a JWT signing secret (x2), and Grafana/Prometheus
basic-auth credentials — are still valid *today* is not something I can
determine from the repository alone, and I did not test any of them against
a live service (I don't have the plaintext — I ran everything with
`--redact=100` — and testing a possibly-real credential against a live
system without your explicit sign-off is exactly the kind of decision your
instructions asked me to bring to you rather than automate). The fact that
current `main` has since parameterized these files is evidence someone
*touched* this area, but it doesn't by itself prove the underlying secret
values were rotated at the service that issued them.

**I'm stopping here per your instructions** ("if any finding is a
genuinely live... credential: STOP... credential rotation is a decision I
need to make explicitly" and "do not add anything to `.gitleaksignore` you
haven't individually verified"). I have not confirmed any of the 7 as
definitely live, but I also can't confirm any as definitely dead — so I'm
not writing `.gitleaksignore` entries or touching Part 3 until you tell me
whether these five secrets (PAT, RAGBIO key, 2x JWT secret, Grafana/
Prometheus creds — all dated 2026-05-22 through 2026-05-28) have been
rotated since. If you confirm rotation/dead, I'll write the five individual
justification lines and proceed to Part 3. If any turns out to still be
live, tell me and I'll stop there entirely per your instructions.

## Part 3 — Not started (blocked on Part 2 above).

## Part 4 — PR #67

Independently verified before merging (didn't just take the task description
on faith): `gh pr view 67 --json files,commits,mergeable` shows a single-file,
7-line, 0-deletion, comment-only addition to `docker-compose.yml` (a `TODO`
above the `rag` service's now-unused `GITHUB_TOKEN`/`RAGBIO_API_KEY` build
args, referencing `omnibioai-rag#25`), `mergeable: MERGEABLE`, diff matches
its own description exactly. No code/behavior change, no secrets touched.

Merged separately — see below.
77 changes: 77 additions & 0 deletions SECRET_SCAN_RESOLUTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Secret-Scan CI Resolution (Issue #63)

Follow-up to `SECRET_SCAN_CI_INVESTIGATION.md`, which diagnosed the root
cause (Part 1) and left Part 2 blocked on live-credential determinations
only the repo owner could make. This document records those
determinations and the resulting fix.

## Root cause (recap)

`actions/checkout@v4` with `fetch-depth: 0` force-updates every tag
(`+refs/tags/*`). The `v0.2.0` / `v0.2.0-beta` tags were never advanced
past a pre-history-rewrite snapshot, so CI's `gitleaks git /repo` (which
walks every ref, not just `main`) permanently rescans two abandoned tag
tips instead of `main`'s real, clean history. `main` itself has 0 leaks in
389 commits.

## Live-credential determinations (owner-verified)

Each of the 7 findings was individually checked — no blanket
justification. Methods used:

| # | Commit | File:Line | Rule | Status | Verification method |
|---|--------|-----------|------|--------|----------------------|
| 1 | `979ba0c0c` | `.env:15` | github-fine-grained-pat | Rotated/revoked | Absent from current GitHub fine-grained token list entirely |
| 2 | `cced81dac` | `.env:8` | generic-api-key (RAGBIO_API_KEY) | Rotated | Direct value comparison: current live value differs from leaked value |
| 3 | `981226817` | `docker-compose.yml:232` | jwt | Dead | JWT payload decoded from leaked token; `exp` ≈ 2026-05-25, already past |
| 4 | `11c868fb5` | `docker-compose.release.yml:213` | jwt | Dead | JWT payload decoded from leaked token; `exp` ≈ 2026-05-25, already past |
| 5-7 | `975460f35` | `DEPLOYMENT.md:232,317,320` | curl-auth-user (Grafana/Prometheus) | Different/safe | Direct comparison: current `GF_ADMIN_PASSWORD` is not the leaked `omnibioai` default |

Findings 3-7 are also byte-identical content (same author, date, diff) to
commits already covered by pre-existing `.gitleaksignore` entries
(`8e35ae534a8`, `9a12a77d00b`, `4f94b88909f`) — those were the correct
main-ancestor commits before the history rewrite changed every descendant
hash. The tag-only commits above are the same secrets under their
pre-rewrite hashes, which is why the old entries didn't suppress them.

Note: the RAGBIO_API_KEY's current live value was independently rotated a
second time during this investigation, for an unrelated reason (accidental
exposure in a separate chat session). That second rotation isn't reflected
in `.gitleaksignore`'s justification comment — it's not something this
specific git-history finding required, and is mentioned here only for the
record.

## Fix applied

Added 7 individually-justified entries to `.gitleaksignore` (see that
file's comments for the per-entry rationale and evidence), one per
fingerprint above. Kept the 5 pre-existing entries as-is.

## Verification

1. Re-fetched with CI's exact refspec
(`+refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*`) to rule
out the stale-local-tag false-clean failure mode described in Part 1 of
the investigation doc. Confirmed local `v0.2.0` / `v0.2.0-beta` already
matched `origin` exactly (no drift this time).
2. Ran a fresh, unfiltered `gitleaks git` scan (no ignore file) directly
against the repo to pull real fingerprints/line numbers from the tool
itself rather than transcribing them — all 7 matched the investigation
doc and the table above exactly.
3. Ran `scripts/secret-scan.sh` (the actual CI step) with the updated
`.gitleaksignore`: `450 commits scanned`, **no leaks found**, exit 0.

## Artifacts

- `gitleaks-report.json` (untracked, `[]` — a clean-`main`-only scan with
zero findings) deleted; nothing in it needed preserving.
- `SECRET_SCAN_CI_INVESTIGATION.md` and this file committed as the
permanent record of the investigation and resolution.

## Remaining steps

- Open a PR on `fix/gitleaks-issue-63-tag-history-secrets` referencing
issue #63, confirm CI goes green there.
- Comment on issue #63 with this resolution once the PR is up.
- Close issue #63 after the PR merges (not before — merge is a human
decision, not automated here).
Loading