Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions docs/ADMIN_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,14 @@ individually**, including the one that runs the LRM binary itself, each with its
"used_memory_mib": 38000
}
],
"unattributed": [
{
"device_number": 1,
"uid": 100000,
"started_at": "2026-07-24T11:00:00+00:00",
"used_memory_mib": 24000
}
],
"devices": [
{"device_number": 0, "peak_utilization_percent": 87}
]
Expand All @@ -284,10 +292,15 @@ Each run takes that long to finish.
`processes[].used_memory_mib` is what that user has allocated on that device — a per-user
total rather than the device's, so nobody else's processes are counted into it.

`unattributed` lists processes whose real UID could not be resolved to a username on this
host (container UIDs, user-namespace remapping). Their owner is unknown, yet they still
count as evidence that the GPU is occupied. Reports from an older `gpu-usage-reporter`
that writes no such field keep parsing.

Where utilization cannot be read — GPUs that return `[N/A]` for `utilization.gpu`, or an
older `gpu-usage-reporter` that writes no `devices` — the field stays empty. Empty means
"whether it was computing cannot be asked", not "it was not computing", and those GPUs are
judged the way they always were: by whether the owner has processes on them. A missing
judged the way they always were: by whether processes sit on them. A missing
`used_memory_mib` works the same way: the amount goes unmentioned, the judgement still runs.

**Enabling the feature on the main binary side:** the reconciliation loop (comparing
Expand All @@ -300,7 +313,7 @@ the same shared directory used by `gpu-usage-reporter` above to enable it:
| `GPU_USAGE_REPORTS_DIR` | (unset = feature disabled) | Shared directory read by `SharedFileResourceUsageObserver` |
| `GPU_USAGE_MAX_STALENESS_SECS` | `300` | How old a report can be before it's ignored |
| `UNRESERVED_USAGE_THRESHOLD_SECS` | `600` | How long unreserved usage must continue before a proposal is sent |
| `IDLE_RESERVATION_THRESHOLD_SECS` | `1800` | How long a reservation must go without any process from its owner before the owner is told. Reservations with less time left than this are left alone |
| `IDLE_RESERVATION_THRESHOLD_SECS` | `1800` | How long a reservation must go without any observed usage before the owner is told. Reservations with less time left than this are left alone |
| `IDLE_HELD_GPU_THRESHOLD_SECS` | `3600` | How long a GPU must be held without computation before the owner is told |
| `COMPUTING_GPU_UTILIZATION_PERCENT` | `5` | The utilization at or above which a GPU counts as computing. Setting it to `0` effectively turns off the held-without-computing check |
| `IDLE_HELD_GPU_NOTICES` | `observe` | Whether a GPU held without computation is reported to its owner (`notify`) or only counted in the logs (`observe`) |
Expand All @@ -310,10 +323,16 @@ the same shared directory used by `gpu-usage-reporter` above to enable it:
**On reservations held without computation:** a process sitting on a GPU is not the same as
computation running on it. Where memory is allocated and then left waiting — a resident
inference server, an open notebook, a stalled training job — the GPU is unavailable to
everyone else while nothing moves forward. If a GPU carrying the owner's processes does not
everyone else while nothing moves forward. If a GPU carrying processes does not
reach `COMPUTING_GPU_UTILIZATION_PERCENT` for `IDLE_HELD_GPU_THRESHOLD_SECS`, the owner gets
a DM.

The judgement does not ask whose processes they are. Container workloads (Docker and the
like) carry UIDs that cannot be traced back to the owner, yet computation on the reserved
device still counts as the reservation being used, and an allocation left behind still
counts as holding it. An OS-username link (`/link-user`) is therefore not a precondition
for this judgement; matching identities is left to unauthorized-usage detection.

GPUs are judged one at a time. One of eight cards computing says nothing about the other
seven. Where only some are at rest, the notice names which ones.

Expand Down Expand Up @@ -356,7 +375,7 @@ scopes for the DM to work.
**Checking that DMs arrive, without involving anyone else:** one reservation of your own is
enough to walk the whole path.

1. Link your own OS username with `/link-user`.
1. Check that your Slack account and email address are linked (`/register-calendar`).
2. Reserve one GPU in your own name for a couple of hours (one nobody else is using).
3. Start with `IDLE_RESERVATION_THRESHOLD_SECS=60`, `IDLE_HELD_GPU_THRESHOLD_SECS=60`, and
`IDLE_HELD_GPU_NOTICES=notify`.
Expand Down
24 changes: 21 additions & 3 deletions docs/ADMIN_GUIDE_ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,12 +257,25 @@ cron登録する必要があります。このバイナリはリリースアー
"used_memory_mib": 38000
}
],
"unattributed": [
{
"device_number": 1,
"uid": 100000,
"started_at": "2026-07-24T11:00:00+00:00",
"used_memory_mib": 24000
}
],
"devices": [
{"device_number": 0, "peak_utilization_percent": 87}
]
}
```

`unattributed`は、実UIDをこのホストのユーザー名に解決できなかったプロセスです
(コンテナ内のUID、user namespaceでずらされたUID等)。持ち主は分かりませんが、
そのGPUが使われていることの証拠として数えられます。この欄を書かない古い
`gpu-usage-reporter`のレポートも読み続けられます。

`generated_at`から一定時間(既定5分想定)経過したファイルは古いデータとみなされ無視されます。
cronが停止した場合に古い利用状況を「今も使用中」と誤判定しないための仕組みです。

Expand All @@ -289,7 +302,7 @@ cronが停止した場合に古い利用状況を「今も使用中」と誤判
| `GPU_USAGE_REPORTS_DIR` | (未設定=機能無効) | `SharedFileResourceUsageObserver`が読み取る共有ディレクトリ |
| `GPU_USAGE_MAX_STALENESS_SECS` | `300` | レポートを無視し始める経過時間(秒) |
| `UNRESERVED_USAGE_THRESHOLD_SECS` | `600` | 未予約利用を提案対象とみなす継続時間の閾値(秒) |
| `IDLE_RESERVATION_THRESHOLD_SECS` | `1800` | 予約者本人のプロセスを観測できない予約を知らせるまでの時間(秒)。残り時間がこれに満たない予約には知らせない |
| `IDLE_RESERVATION_THRESHOLD_SECS` | `1800` | 利用をひとつも観測できない予約を知らせるまでの時間(秒)。残り時間がこれに満たない予約には知らせない |
| `IDLE_HELD_GPU_THRESHOLD_SECS` | `3600` | GPUを押さえたまま計算が走らない予約を知らせるまでの時間(秒) |
| `COMPUTING_GPU_UTILIZATION_PERCENT` | `5` | 計算が走っているとみなす稼働率(%)。`0`にすると押さえたまま計算していない予約の検知が実質的に止まる |
| `IDLE_HELD_GPU_NOTICES` | `observe` | 押さえたまま計算していない予約を、`notify`(予約者に知らせる)か`observe`(ログに数えるだけ)か |
Expand All @@ -299,9 +312,14 @@ cronが停止した場合に古い利用状況を「今も使用中」と誤判
**押さえたまま計算していない予約について**: プロセスが乗っていることは、計算が走っていることを
意味しません。メモリだけを確保して待機させる使い方(常駐させた推論サーバー、開いたままの
ノートブック、止まった学習ジョブ)では、GPUは他の人が使えないまま何も進んでいません。
予約者本人のプロセスが乗っているGPUで`COMPUTING_GPU_UTILIZATION_PERCENT`以上の稼働率が
プロセスが乗っているGPUで`COMPUTING_GPU_UTILIZATION_PERCENT`以上の稼働率が
`IDLE_HELD_GPU_THRESHOLD_SECS`のあいだ一度も出なければ、予約者へDMを送ります。

判定は誰のプロセスかを問いません。Docker等のコンテナ実行ではプロセスの名義を予約者に
辿れませんが、予約が押さえているデバイスの上で計算が走っていれば使われている、確保だけが
残っていれば押さえたまま、と読みます。このため予約者のOSユーザー名の紐付け(`/link-user`)は
この判定の前提ではありません。名義の照合は無断使用の検知だけが行います。

判定はGPU1台ずつ行います。8枚のうち1枚で計算が走っていることは、残りの7枚が使われている
ことを意味しません。一部だけが休んでいる場合は、どのGPUのことなのかを添えて知らせます。

Expand Down Expand Up @@ -341,7 +359,7 @@ Slackアカウントも紐付いたメールアドレスにリンクされてい

**DMが届くことを、人を巻き込まずに確かめる**: 管理者自身の予約1つで一巡させられます。

1. 自分のOSユーザー名を`/link-user`で紐付ける
1. 自分のSlackアカウントとメールアドレスが紐付いていること(`/register-calendar`)を確かめる
2. 自分名義でGPUを1枚、2時間ほど予約する(他の人が使っていないもの)
3. `IDLE_RESERVATION_THRESHOLD_SECS=60`・`IDLE_HELD_GPU_THRESHOLD_SECS=60`・
`IDLE_HELD_GPU_NOTICES=notify`で起動する
Expand Down
5 changes: 3 additions & 2 deletions docs/USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,12 +186,13 @@ keep others waiting. If you still intend to use it, press "✅ Still using it" a
though the quiet lasts a while rather than forever. If it still goes unused after that, you
will hear about it again.

Whose processes they are does not matter: computation running inside Docker or any other
container still counts as your reservation being used.

Some reservations are never reported:

- Reservations on a server that cannot be observed (without a report, whether it is in use
is unknown)
- Reservations whose owner has no OS username linked via `/link-user` (your processes cannot
be told apart from anyone else's)
- Meeting room reservations
- Reservations that are about to end

Expand Down
4 changes: 3 additions & 1 deletion docs/USER_GUIDE_ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,10 +170,12 @@ GPUの実利用監視が有効になっている場合、予約したGPUがし
「✅ まだ使う」を押してそのまま続けてください。ただし黙るのは一定時間です。そのあとも
使われていなければ、改めてお知らせが届きます。

誰のプロセスかは判定に影響しません。Dockerなどのコンテナの中で使っていても、予約した
GPUで計算が走っていれば使われていると判定されます。

次の予約はお知らせの対象外です。

- 監視できていないサーバーの予約(レポートが届いていない場合、使われているかどうかを判断できません)
- `/link-user` でOSユーザー名が紐付いていない予約(あなたのプロセスをあなたのものと見分けられません)
- 会議室の予約
- まもなく終わる予約

Expand Down
1 change: 1 addition & 0 deletions src/application/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ use std::fmt;
///
/// インフラストラクチャ層、ドメイン層、およびユースケース固有のエラーをラップする。
#[derive(Debug)]
#[non_exhaustive]
pub enum ApplicationError {
/// リポジトリ操作中に発生したエラー
Repository(RepositoryError),
Expand Down
Loading
Loading