🤖 Generated by the Agentic Engineer
Evidence
Longhorn detects replica data left behind on disk and files an Orphan record for each directory, but
never deletes any of it: the orphanResourceAutoDeletion setting ships empty, so reclamation is
opt-in and was never enabled here.
Observed in prod: one storage worker was holding 14 orphaned replica directories, sat at 24%
free, and went DiskPressure. Because Longhorn requires replicas on distinct nodes and this cluster
has only three storage nodes, two databases were then unable to rebuild their third replica at
all — they stayed degraded indefinitely rather than recovering on their own.
Affected audience and impact
Every workload with Longhorn-backed storage. The failure is silent and cumulative: each autoscaler
scale-down and each worker roll adds more orphaned directories, so free space decays until a node
crosses the threshold and stops accepting replicas. The first visible symptom is a database stuck
degraded, which reads like a database fault rather than a storage-capacity one.
Expected behaviour
Orphaned replica data is reclaimed automatically, so disk space returns without operator action.
Orphaned instances are deliberately out of scope — only disk-consuming replica data is swept.
Acceptance criteria
Size: small — one chart value, plus the render verification.
Relationship to #3180
Filed because #3180 is a different failure mode and must stay open. #3180 is a stale iSCSI node
record poisoning a node's iSCSI database so every Longhorn engine startup on that node fails; its
stated ask is iSCSI record detection and cleanup, and its acceptance criteria are all about
iscsiadm -m node -o show. Orphaned replica-data reclamation neither detects nor removes an iSCSI
record, so it cannot satisfy those criteria. The two were previously linked by a Fixes #3180 in
PR #3181; that link is corrected to point here, leaving #3180 open for its own work.
Evidence
Longhorn detects replica data left behind on disk and files an
Orphanrecord for each directory, butnever deletes any of it: the
orphanResourceAutoDeletionsetting ships empty, so reclamation isopt-in and was never enabled here.
Observed in prod: one storage worker was holding 14 orphaned replica directories, sat at 24%
free, and went
DiskPressure. Because Longhorn requires replicas on distinct nodes and this clusterhas only three storage nodes, two databases were then unable to rebuild their third replica at
all — they stayed degraded indefinitely rather than recovering on their own.
Affected audience and impact
Every workload with Longhorn-backed storage. The failure is silent and cumulative: each autoscaler
scale-down and each worker roll adds more orphaned directories, so free space decays until a node
crosses the threshold and stops accepting replicas. The first visible symptom is a database stuck
degraded, which reads like a database fault rather than a storage-capacity one.
Expected behaviour
Orphaned replica data is reclaimed automatically, so disk space returns without operator action.
Orphaned instances are deliberately out of scope — only disk-consuming replica data is swept.
Acceptance criteria
orphanResourceAutoDeletionincludesreplica-datain the rendered Longhorn configurationthe change is not a silent no-op
Size: small — one chart value, plus the render verification.
Relationship to #3180
Filed because #3180 is a different failure mode and must stay open. #3180 is a stale iSCSI node
record poisoning a node's iSCSI database so every Longhorn engine startup on that node fails; its
stated ask is iSCSI record detection and cleanup, and its acceptance criteria are all about
iscsiadm -m node -o show. Orphaned replica-data reclamation neither detects nor removes an iSCSIrecord, so it cannot satisfy those criteria. The two were previously linked by a
Fixes #3180inPR #3181; that link is corrected to point here, leaving #3180 open for its own work.