Skip to content

feat!: judge idle reservations by the reserved device, not process identity - #161

Merged
KinjiKawaguchi merged 5 commits into
mainfrom
fix/device-first-idle-detection
Sep 21, 2026
Merged

KinjiKawaguchi merged 5 commits into
mainfrom
fix/device-first-idle-detection

Conversation

@KinjiKawaguchi

@KinjiKawaguchi KinjiKawaguchi commented Sep 17, 2026

Copy link
Copy Markdown
Member

Why

Running a workload inside Docker made the idle-reservation watch nag its owner repeatedly, because usage attribution rested on a single chain — PID → real UID → getent → username → owner — that breaks under containers, user namespaces, and shared accounts. The breakage was swallowed silently: the reporter dropped unresolvable processes, and the judgement read "no process attributable to the owner" as "absent".

Closes #160

What

  • Judge idleness device-first. judge_reservation_activity now reads what happens on the reserved device — utilization and the presence of any usage — without asking whose processes they are. A reservation already binds the device to the person responsible; matching identities stays with unauthorized-usage detection. The OS-username link is no longer a precondition for judging, so owners without one now receive idle notices too.
  • Report unattributed processes. gpu-usage-reporter reports processes whose UID cannot be resolved as unattributed entries (with the raw UID) instead of dropping them, and the shared-file observer carries them into the snapshot. The field is additive and optional in both directions: old reports keep parsing, old servers ignore the new field.
  • Quiet handling for unknown DM recipients. An owner with no Slack link used to surface as a SendFailure on every poll. NotificationError::RecipientUnknown now tells "we do not know where to send this" apart from "sending broke"; idle detection counts it quietly and backs off for the usual silence window.
  • Guides updated accordingly (judgement wording, report schema, verification steps).

Behavior changes

  • Someone else computing on a reserved GPU no longer triggers a misleading "not in use" notice to the owner (the squatter is still handled by the unauthorized-usage flow).
  • Owners who never linked an OS username were previously never notified; they now are.
  • "Held without computing" covers any allocation on the reserved device, not only the owner's own processes.

How

Breaking for the library API — the next release becomes 2.0.0: DetectIdleReservationsUseCase loses its IdentityLinkRepository type parameter and constructor argument (its judgement also becomes synchronous), judge_reservation_activity drops owner_identities, GpuUsageReport gains a required field, and NotificationError gains a variant. Riding the same major: all public error enums are now #[non_exhaustive], so future failure kinds ship as minor releases; external matches need a wildcard arm.

Nothing operational breaks: no env var changes, the report JSON schema is additive, and reporters and the server can be updated in either order. ObservationSnapshot gains unattributed_usages, kept as a parallel collection so the reconcile flow is untouched.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings September 17, 2026 08:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

judge_reservation_activity can still misclassify a reservation as Absent when utilization indicates computing but no process entries are present (e.g., short-lived work between utilization sampling and process snapshot), which risks reintroducing false idle notices.

Pull request overview

This PR updates idle-reservation detection to judge “in use / held / absent” based on what is happening on the reserved GPU devices (utilization + any observed usage), rather than attributing processes to the reservation owner (which breaks under containers / namespaces). It also extends the GPU usage report schema to carry “unattributed” usages (UIDs that cannot be resolved to OS usernames) and refines notification error handling so unknown DM recipients don’t count as repeated delivery failures.

Changes:

  • Make reservation activity judgement device-first and include unattributed usages in snapshots and judgement.
  • Extend gpu-usage-reporter + shared-file observer schema with unattributed usage entries to avoid silently dropping unresolvable UIDs.
  • Introduce NotificationError::RecipientUnknown and adjust idle-notice handling/backoff + docs accordingly.
File summaries
File Description
src/lib.rs Re-exports the new UnattributedUsageEntry in the public prelude.
src/infrastructure/slack_direct_message.rs Classifies missing IdentityLink/Slack link as RecipientUnknown instead of SendFailure.
src/infrastructure/resource_usage_observer/shared_file.rs Adds unattributed parsing/serialization and threads unattributed usages into ObservationSnapshot.
src/infrastructure/resource_usage_observer/mod.rs Re-exports UnattributedUsageEntry from the shared-file observer module.
src/infrastructure/idle_reservation_notifier/slack.rs Updates idle-notice wording to be owner-agnostic (“利用を確認できていません”).
src/infrastructure/idle_reservation_notifier/mock.rs Adds mock behavior for RecipientUnknown to test quiet/backoff behavior.
src/domain/services/resource_usage/reservation_activity.rs Changes reservation activity judgement to be identity-agnostic and include unattributed usages.
src/domain/ports/resource_usage_observer.rs Introduces UnattributedUsage + ObservationSnapshot::with_unattributed_usages/accessor.
src/domain/ports/notifier.rs Adds NotificationError::RecipientUnknown and Display formatting.
src/domain/ports/mod.rs Re-exports UnattributedUsage through the domain ports module.
src/domain/ports/idle_reservation_notifier.rs Updates idle evidence semantics to be about “any usage” on reserved GPUs.
src/bin/lab-resource-manager.rs Updates use case wiring due to DetectIdleReservationsUseCase signature change.
src/bin/gpu-usage-reporter.rs Emits unattributed entries (device_number, uid, started_at, used_memory_mib) instead of dropping unknown UIDs.
src/application/usecases/detect_idle_reservations.rs Removes identity-link dependency for judging; treats RecipientUnknown quietly with normal silence backoff.
src/application/usecases/detect_idle_reservations_tests.rs Updates tests for device-first judgement and adds unattributed + recipient-unknown scenarios.
docs/USER_GUIDE.md Documents that containerized compute still counts as “in use” regardless of process identity.
docs/USER_GUIDE_ja.md Japanese guide updates matching the device-first judgement behavior.
docs/ADMIN_GUIDE.md Documents unattributed schema and updates idle threshold semantics/verification steps.
docs/ADMIN_GUIDE_ja.md Japanese admin guide updates for unattributed schema and device-first judgement behavior.
Review details

Suppressed comments (2)

src/domain/services/resource_usage/reservation_activity.rs:111

  • judge_reservation_activity returns Absent when no process-based usage is observed, even if GPU activity shows the reserved device was computing during the sampling window. Because gpu-usage-reporter samples utilization over a window and only then snapshots processes, short-lived or just-finished workloads can legitimately produce devices[].peak_utilization_percent without any processes/unattributed entries, and this path would misclassify them as absent and potentially trigger an idle notice.
    let occupied = occupied_gpus(reserved, snapshot);

    if occupied.is_empty() {
        return ReservationActivity::Absent;
    }

src/bin/gpu-usage-reporter.rs:382

  • resolve_username is invoked once per observed process, which can spawn many getent subprocesses when multiple processes share the same UID. Caching UID→username (including negative lookups) within the process would avoid redundant getent calls and reduce runtime overhead.
/// UIDをこのホストのOSユーザー名に解決する
///
/// コンテナ内のUIDやuser namespaceでずらされたUIDは、このホストのアカウントに
/// 対応せず解決できない。それは利用がないことではなく、持ち主が分からないことを意味する。
fn resolve_username(uid: u32) -> Option<String> {
    let output = Command::new("getent")
        .args(["passwd", &uid.to_string()])
        .output()
        .ok()?;
    if !output.status.success() {
        return None;
    }

    let stdout = String::from_utf8(output.stdout).ok()?;
    stdout.split(':').next().map(str::to_string)
}
  • Files reviewed: 19/19 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@KinjiKawaguchi
KinjiKawaguchi force-pushed the fix/device-first-idle-detection branch from 7d82280 to 413144a Compare September 18, 2026 02:32
@KinjiKawaguchi KinjiKawaguchi changed the title feat!: judge idle reservations by the reserved device, not process identity feat: judge idle reservations by the reserved device, not process identity Sep 18, 2026
@KinjiKawaguchi
KinjiKawaguchi force-pushed the fix/device-first-idle-detection branch from 413144a to a24d67a Compare September 21, 2026 08:44
@KinjiKawaguchi KinjiKawaguchi changed the title feat: judge idle reservations by the reserved device, not process identity feat!: judge idle reservations by the reserved device, not process identity Sep 21, 2026
Copilot AI review requested due to automatic review settings September 21, 2026 09:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

judge_reservation_activity can still classify a reservation as Absent even when utilization sampling indicates computation (utilization-only evidence is ignored when no process entries are present).

Review effort: Lite
Findings: None

KinjiKawaguchi and others added 5 commits September 21, 2026 18:36
… process identity

Idle detection used to require every observed process to be attributed to
the reservation owner (PID -> real UID -> username -> owner). Containers,
user namespaces, and shared accounts break that chain, so owners computing
inside Docker were repeatedly told their reservation was not in use.

A reservation already binds the device to the person responsible for it.
Judge InUse / HeldWithoutComputing / Absent from what runs on the reserved
device, whoever it belongs to; matching identities remains the job of
unauthorized-usage detection. This also introduces unattributed usage as a
first-class observation, and drops the OS-username link as a precondition
for judging (owners without one now receive idle notices too).

BREAKING CHANGE: DetectIdleReservationsUseCase loses its
IdentityLinkRepository type parameter and constructor argument, and
judge_reservation_activity drops its owner_identities parameter.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…instead of dropping them

The reporter silently dropped any process whose UID getent could not
resolve (container UIDs, user-namespace remapping), which made the GPU it
occupies look untouched. Report such processes as unattributed entries
with their raw UID, and carry them through the shared-file observer into
the observation snapshot. The new report field is additive and optional,
so old reports keep parsing and old servers ignore the new field.

BREAKING CHANGE: GpuUsageReport gains a required unattributed field, so
struct-literal construction needs updating. Deserialization of reports
written without the field is unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… a delivery failure

Owners judged idle but with no Slack link resolved to a SendFailure on
every poll, polluting the failure count and the error log with a condition
no retry can cure. Distinguish "we do not know where to send this"
(RecipientUnknown) from "sending broke", and have idle detection count it
quietly and back off for the usual silence window.

BREAKING CHANGE: NotificationError gains a RecipientUnknown variant, so
exhaustive matches need a new arm.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OS-username link is no longer a precondition for idle notices, the
judgement no longer asks whose processes sit on the reserved device, and
reports now carry an unattributed section for processes whose UID cannot
be resolved (container workloads).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Error classifications are open sets: the ways notification, observation,
or configuration can fail grow with the infrastructure. Leaving these
enums exhaustively matchable makes every new failure kind a major
version, as adding RecipientUnknown just demonstrated. Mark all public
error enums non_exhaustive while the 2.0.0 window is open, so future
variants ship as minor releases.

BREAKING CHANGE: matches on public error enums outside this crate now
need a wildcard arm.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings September 21, 2026 09:37
@KinjiKawaguchi
KinjiKawaguchi force-pushed the fix/device-first-idle-detection branch from b7a8203 to 31f52cf Compare September 21, 2026 09:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It introduces breaking API changes and modifies core idle/notification behavior across multiple layers (reporter, observer, domain logic, and docs), warranting final human review.

Review effort: Lite
Findings: None

@KinjiKawaguchi
KinjiKawaguchi merged commit 6141e74 into main Sep 21, 2026
11 checks passed
@KinjiKawaguchi
KinjiKawaguchi deleted the fix/device-first-idle-detection branch September 21, 2026 09:40
KinjiKawaguchi added a commit that referenced this pull request Sep 21, 2026
#162)

## Why

The crate was deliberately published to crates.io from the very
beginning (#2), but publishing was a manual step. When release-please
automation landed (#77), only the GitHub Release artifacts were wired
in, and the manual publish quietly stopped happening: crates.io has been
stuck at **1.1.0 (2026-01-27)** while the repository moved on to 1.8.0 —
with the README badge advertising the stale version.

Resuming publication also revives a promise: the crate is consumed as a
library, so a version bump that does not cover its API changes would
break downstream builds on `cargo update`. Nothing in the pipeline
verified that promise — the build jobs check that *our binaries* build,
and `cargo publish`'s verify step checks that *the crate compiles*,
neither of which is API compatibility.

## What

Adds a `publish-crate` job to the release workflow that runs whenever
release-please cuts a release:

1. **`cargo-semver-checks`** compares the tree against the last version
published on crates.io and fails if the version bump does not cover the
API changes.
2. **`cargo publish`** runs only if the check passes.

The job runs alongside the binary build; a failure there does not block
the tarball upload, and vice versa.

## Verified locally

- `cargo publish --dry-run` passes (packaging and verify build).
- `cargo semver-checks --baseline-version 1.1.0` on the current tree
(1.8.0) **fails with 12 major-level breaks** — the accumulated 1.2–1.8
API drift, correctly caught.
- The same check with the version set to 2.0.0 passes — a major bump
settles the accumulated drift.

Since #161 carries `feat!:` commits, the next release will be **2.0.0**,
so the gate passes on its first run and enforces from then on.

## Setup required before merging

The job needs a `CARGO_REGISTRY_TOKEN` repository secret: a crates.io
API token with the `publish-update` scope (ideally scoped to the
`lab-resource-manager` crate), created at crates.io → Account Settings →
API Tokens. Until the secret exists, the job will fail on the next
release — everything else in the workflow is unaffected.

Versions 1.2.0–1.8.0 stay unpublished; crates.io catches up at 2.0.0.

## Follow-up (not this PR)

A PR-time semver check (in code-quality.yml) would catch a missing `!`
when it is still cheap to fix — but it only becomes meaningful once
2.0.0 exists as the baseline, so it is left for after the first release.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
KinjiKawaguchi pushed a commit that referenced this pull request Sep 21, 2026
🤖 I have created a release *beep* *boop*
---


##
[2.0.0](v1.8.0...v2.0.0)
(2026-09-21)


### ⚠ BREAKING CHANGES

* google_calendar_mappings.json is no longer read and the
GOOGLE_CALENDAR_MAPPINGS_FILE environment variable is gone (both are
ignored if left in place; reservation ids copied from pre-v1.5.1 event
descriptions stop resolving). In the library API,
GoogleCalendarUsageRepository::new no longer takes the mappings path.
* existing resources.toml files stop loading until calendar_id entries
are moved into the [storage] section (see the migration guide). In the
library API, load_config returns a (ResourceConfig, StorageConfig) pair,
ServerConfig and RoomConfig lose their calendar_id field, and
GoogleCalendarUsageRepository::new takes the StorageConfig.
* TemplateConfig gains a conflict_item field, so struct-literal
construction outside this crate needs updating.
* DetectIdleReservationsUseCase loses its IdentityLinkRepository type
parameter and constructor argument; judge_reservation_activity drops its
owner_identities parameter; GpuUsageReport gains a required unattributed
field; NotificationError gains a RecipientUnknown variant; public error
enums are non_exhaustive, so external matches need a wildcard arm.

### Features

* improve how multiple reservation conflicts are presented
([#164](#164))
([bbfb86b](bbfb86b))
* judge idle reservations by the reserved device, not process identity
([#161](#161))
([6141e74](6141e74))


### Code Refactoring

* drop the id-mapping file for calendar events
([#166](#166))
([88570ee](88570ee))
* separate the storage mapping from resource definitions
([#165](#165))
([cbc3fdc](cbc3fdc))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Idle detection reads unattributable workloads (e.g. containers) as absence

2 participants