feat!: judge idle reservations by the reserved device, not process identity - #161
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
judge_reservation_activity can still misclassify a reservation as Absent when utilization indicates computing but no process entries are present (e.g., short-lived work between utilization sampling and process snapshot), which risks reintroducing false idle notices.
Pull request overview
This PR updates idle-reservation detection to judge “in use / held / absent” based on what is happening on the reserved GPU devices (utilization + any observed usage), rather than attributing processes to the reservation owner (which breaks under containers / namespaces). It also extends the GPU usage report schema to carry “unattributed” usages (UIDs that cannot be resolved to OS usernames) and refines notification error handling so unknown DM recipients don’t count as repeated delivery failures.
Changes:
- Make reservation activity judgement device-first and include unattributed usages in snapshots and judgement.
- Extend
gpu-usage-reporter+ shared-file observer schema withunattributedusage entries to avoid silently dropping unresolvable UIDs. - Introduce
NotificationError::RecipientUnknownand adjust idle-notice handling/backoff + docs accordingly.
File summaries
| File | Description |
|---|---|
| src/lib.rs | Re-exports the new UnattributedUsageEntry in the public prelude. |
| src/infrastructure/slack_direct_message.rs | Classifies missing IdentityLink/Slack link as RecipientUnknown instead of SendFailure. |
| src/infrastructure/resource_usage_observer/shared_file.rs | Adds unattributed parsing/serialization and threads unattributed usages into ObservationSnapshot. |
| src/infrastructure/resource_usage_observer/mod.rs | Re-exports UnattributedUsageEntry from the shared-file observer module. |
| src/infrastructure/idle_reservation_notifier/slack.rs | Updates idle-notice wording to be owner-agnostic (“利用を確認できていません”). |
| src/infrastructure/idle_reservation_notifier/mock.rs | Adds mock behavior for RecipientUnknown to test quiet/backoff behavior. |
| src/domain/services/resource_usage/reservation_activity.rs | Changes reservation activity judgement to be identity-agnostic and include unattributed usages. |
| src/domain/ports/resource_usage_observer.rs | Introduces UnattributedUsage + ObservationSnapshot::with_unattributed_usages/accessor. |
| src/domain/ports/notifier.rs | Adds NotificationError::RecipientUnknown and Display formatting. |
| src/domain/ports/mod.rs | Re-exports UnattributedUsage through the domain ports module. |
| src/domain/ports/idle_reservation_notifier.rs | Updates idle evidence semantics to be about “any usage” on reserved GPUs. |
| src/bin/lab-resource-manager.rs | Updates use case wiring due to DetectIdleReservationsUseCase signature change. |
| src/bin/gpu-usage-reporter.rs | Emits unattributed entries (device_number, uid, started_at, used_memory_mib) instead of dropping unknown UIDs. |
| src/application/usecases/detect_idle_reservations.rs | Removes identity-link dependency for judging; treats RecipientUnknown quietly with normal silence backoff. |
| src/application/usecases/detect_idle_reservations_tests.rs | Updates tests for device-first judgement and adds unattributed + recipient-unknown scenarios. |
| docs/USER_GUIDE.md | Documents that containerized compute still counts as “in use” regardless of process identity. |
| docs/USER_GUIDE_ja.md | Japanese guide updates matching the device-first judgement behavior. |
| docs/ADMIN_GUIDE.md | Documents unattributed schema and updates idle threshold semantics/verification steps. |
| docs/ADMIN_GUIDE_ja.md | Japanese admin guide updates for unattributed schema and device-first judgement behavior. |
Review details
Suppressed comments (2)
src/domain/services/resource_usage/reservation_activity.rs:111
judge_reservation_activityreturnsAbsentwhen no process-based usage is observed, even if GPU activity shows the reserved device was computing during the sampling window. Becausegpu-usage-reportersamples utilization over a window and only then snapshots processes, short-lived or just-finished workloads can legitimately producedevices[].peak_utilization_percentwithout anyprocesses/unattributedentries, and this path would misclassify them as absent and potentially trigger an idle notice.
let occupied = occupied_gpus(reserved, snapshot);
if occupied.is_empty() {
return ReservationActivity::Absent;
}
src/bin/gpu-usage-reporter.rs:382
resolve_usernameis invoked once per observed process, which can spawn manygetentsubprocesses when multiple processes share the same UID. Caching UID→username (including negative lookups) within the process would avoid redundantgetentcalls and reduce runtime overhead.
/// UIDをこのホストのOSユーザー名に解決する
///
/// コンテナ内のUIDやuser namespaceでずらされたUIDは、このホストのアカウントに
/// 対応せず解決できない。それは利用がないことではなく、持ち主が分からないことを意味する。
fn resolve_username(uid: u32) -> Option<String> {
let output = Command::new("getent")
.args(["passwd", &uid.to_string()])
.output()
.ok()?;
if !output.status.success() {
return None;
}
let stdout = String::from_utf8(output.stdout).ok()?;
stdout.split(':').next().map(str::to_string)
}
- Files reviewed: 19/19 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
7d82280 to
413144a
Compare
413144a to
a24d67a
Compare
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
judge_reservation_activity can still classify a reservation as Absent even when utilization sampling indicates computation (utilization-only evidence is ignored when no process entries are present).
Review effort: Lite
Findings: None
… process identity Idle detection used to require every observed process to be attributed to the reservation owner (PID -> real UID -> username -> owner). Containers, user namespaces, and shared accounts break that chain, so owners computing inside Docker were repeatedly told their reservation was not in use. A reservation already binds the device to the person responsible for it. Judge InUse / HeldWithoutComputing / Absent from what runs on the reserved device, whoever it belongs to; matching identities remains the job of unauthorized-usage detection. This also introduces unattributed usage as a first-class observation, and drops the OS-username link as a precondition for judging (owners without one now receive idle notices too). BREAKING CHANGE: DetectIdleReservationsUseCase loses its IdentityLinkRepository type parameter and constructor argument, and judge_reservation_activity drops its owner_identities parameter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…instead of dropping them The reporter silently dropped any process whose UID getent could not resolve (container UIDs, user-namespace remapping), which made the GPU it occupies look untouched. Report such processes as unattributed entries with their raw UID, and carry them through the shared-file observer into the observation snapshot. The new report field is additive and optional, so old reports keep parsing and old servers ignore the new field. BREAKING CHANGE: GpuUsageReport gains a required unattributed field, so struct-literal construction needs updating. Deserialization of reports written without the field is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… a delivery failure Owners judged idle but with no Slack link resolved to a SendFailure on every poll, polluting the failure count and the error log with a condition no retry can cure. Distinguish "we do not know where to send this" (RecipientUnknown) from "sending broke", and have idle detection count it quietly and back off for the usual silence window. BREAKING CHANGE: NotificationError gains a RecipientUnknown variant, so exhaustive matches need a new arm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OS-username link is no longer a precondition for idle notices, the judgement no longer asks whose processes sit on the reserved device, and reports now carry an unattributed section for processes whose UID cannot be resolved (container workloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Error classifications are open sets: the ways notification, observation, or configuration can fail grow with the infrastructure. Leaving these enums exhaustively matchable makes every new failure kind a major version, as adding RecipientUnknown just demonstrated. Mark all public error enums non_exhaustive while the 2.0.0 window is open, so future variants ship as minor releases. BREAKING CHANGE: matches on public error enums outside this crate now need a wildcard arm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7a8203 to
31f52cf
Compare
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
It introduces breaking API changes and modifies core idle/notification behavior across multiple layers (reporter, observer, domain logic, and docs), warranting final human review.
Review effort: Lite
Findings: None
#162) ## Why The crate was deliberately published to crates.io from the very beginning (#2), but publishing was a manual step. When release-please automation landed (#77), only the GitHub Release artifacts were wired in, and the manual publish quietly stopped happening: crates.io has been stuck at **1.1.0 (2026-01-27)** while the repository moved on to 1.8.0 — with the README badge advertising the stale version. Resuming publication also revives a promise: the crate is consumed as a library, so a version bump that does not cover its API changes would break downstream builds on `cargo update`. Nothing in the pipeline verified that promise — the build jobs check that *our binaries* build, and `cargo publish`'s verify step checks that *the crate compiles*, neither of which is API compatibility. ## What Adds a `publish-crate` job to the release workflow that runs whenever release-please cuts a release: 1. **`cargo-semver-checks`** compares the tree against the last version published on crates.io and fails if the version bump does not cover the API changes. 2. **`cargo publish`** runs only if the check passes. The job runs alongside the binary build; a failure there does not block the tarball upload, and vice versa. ## Verified locally - `cargo publish --dry-run` passes (packaging and verify build). - `cargo semver-checks --baseline-version 1.1.0` on the current tree (1.8.0) **fails with 12 major-level breaks** — the accumulated 1.2–1.8 API drift, correctly caught. - The same check with the version set to 2.0.0 passes — a major bump settles the accumulated drift. Since #161 carries `feat!:` commits, the next release will be **2.0.0**, so the gate passes on its first run and enforces from then on. ## Setup required before merging The job needs a `CARGO_REGISTRY_TOKEN` repository secret: a crates.io API token with the `publish-update` scope (ideally scoped to the `lab-resource-manager` crate), created at crates.io → Account Settings → API Tokens. Until the secret exists, the job will fail on the next release — everything else in the workflow is unaffected. Versions 1.2.0–1.8.0 stay unpublished; crates.io catches up at 2.0.0. ## Follow-up (not this PR) A PR-time semver check (in code-quality.yml) would catch a missing `!` when it is still cheap to fix — but it only becomes meaningful once 2.0.0 exists as the baseline, so it is left for after the first release. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
🤖 I have created a release *beep* *boop* --- ## [2.0.0](v1.8.0...v2.0.0) (2026-09-21) ### ⚠ BREAKING CHANGES * google_calendar_mappings.json is no longer read and the GOOGLE_CALENDAR_MAPPINGS_FILE environment variable is gone (both are ignored if left in place; reservation ids copied from pre-v1.5.1 event descriptions stop resolving). In the library API, GoogleCalendarUsageRepository::new no longer takes the mappings path. * existing resources.toml files stop loading until calendar_id entries are moved into the [storage] section (see the migration guide). In the library API, load_config returns a (ResourceConfig, StorageConfig) pair, ServerConfig and RoomConfig lose their calendar_id field, and GoogleCalendarUsageRepository::new takes the StorageConfig. * TemplateConfig gains a conflict_item field, so struct-literal construction outside this crate needs updating. * DetectIdleReservationsUseCase loses its IdentityLinkRepository type parameter and constructor argument; judge_reservation_activity drops its owner_identities parameter; GpuUsageReport gains a required unattributed field; NotificationError gains a RecipientUnknown variant; public error enums are non_exhaustive, so external matches need a wildcard arm. ### Features * improve how multiple reservation conflicts are presented ([#164](#164)) ([bbfb86b](bbfb86b)) * judge idle reservations by the reserved device, not process identity ([#161](#161)) ([6141e74](6141e74)) ### Code Refactoring * drop the id-mapping file for calendar events ([#166](#166)) ([88570ee](88570ee)) * separate the storage mapping from resource definitions ([#165](#165)) ([cbc3fdc](cbc3fdc)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Why
Running a workload inside Docker made the idle-reservation watch nag its owner repeatedly, because usage attribution rested on a single chain — PID → real UID →
getent→ username → owner — that breaks under containers, user namespaces, and shared accounts. The breakage was swallowed silently: the reporter dropped unresolvable processes, and the judgement read "no process attributable to the owner" as "absent".Closes #160
What
judge_reservation_activitynow reads what happens on the reserved device — utilization and the presence of any usage — without asking whose processes they are. A reservation already binds the device to the person responsible; matching identities stays with unauthorized-usage detection. The OS-username link is no longer a precondition for judging, so owners without one now receive idle notices too.gpu-usage-reporterreports processes whose UID cannot be resolved asunattributedentries (with the raw UID) instead of dropping them, and the shared-file observer carries them into the snapshot. The field is additive and optional in both directions: old reports keep parsing, old servers ignore the new field.SendFailureon every poll.NotificationError::RecipientUnknownnow tells "we do not know where to send this" apart from "sending broke"; idle detection counts it quietly and backs off for the usual silence window.Behavior changes
How
Breaking for the library API — the next release becomes 2.0.0:
DetectIdleReservationsUseCaseloses itsIdentityLinkRepositorytype parameter and constructor argument (its judgement also becomes synchronous),judge_reservation_activitydropsowner_identities,GpuUsageReportgains a required field, andNotificationErrorgains a variant. Riding the same major: all public error enums are now#[non_exhaustive], so future failure kinds ship as minor releases; external matches need a wildcard arm.Nothing operational breaks: no env var changes, the report JSON schema is additive, and reporters and the server can be updated in either order.
ObservationSnapshotgainsunattributed_usages, kept as a parallel collection so the reconcile flow is untouched.🤖 Generated with Claude Code