HIFO skips a partially-consumed lot permanently after a large earn-typed event (regression of #73)

[ahalekelly]

Written and filed by Claude Fable (Anthropic AI assistant) at the account owner's request; the investigation and repro below are the AI's work and were not independently reviewed before filing.

Summary

With a feature-based accounting method (HIFO; LOFO should be equally affected), a lot that is partially consumed by a sale is permanently removed from the candidate heap when the next taxable event is an earn-typed transaction whose amount is more than twice the lot's remainder. Every later disposal then skips that lot — even when it is the highest-basis lot available — and consumes cheaper lots instead. The lot's partial-amount bookkeeping still shows the correct positive remainder, and the lot still appears in open positions; it is only the heap that loses it.

This is the same user-visible symptom as #73 (fixed in the pre-heap implementation), reintroduced by the feature-based heap architecture. It is distinct from #150 (sort criterion): here the selection machinery drops a candidate outright, regardless of criterion.

Observed on rp2 1.7.2, Python 3.12, macOS.

Minimal reproduction (4 rows)

One asset, one exchange, method hifo:

id	type	timestamp	amount	spot
b1	BUY	2021-01-01	10	$30
b2	BUY	2021-02-01	10	$20
b3	BUY	2021-03-01	10	$10
s1	SELL	2022-06-01	4	$50
e1	STAKING	2024-01-01	15	$5
s2	SELL	2024-06-01	20	$50

• s1 correctly consumes 4 of the $30 lot (HIFO), leaving a 6-unit remainder.
• e1 is income only — it should not affect which lots are available.
• s2 should consume 6 @ $30 + 10 @ $20 + 4 @ $10 = cost basis 420.

Actual output (hifo_rp2_full_report.ods, Gain/Loss Summary):

2024 | TST | 700.0 | LONG | SELL | 20.0 | 1000.0 | 300.0   <-- basis 300, not 420

The Gain/Loss Detail shows s2 consumed 10 @ $20 + 10 @ $10, skipping the $30 remainder entirely. The 6 units remain in Account Balances / open positions with their original basis, so total basis is conserved — but gains are misstated in every year the lot should have been used (and, when the skipped lot is older, short/long-term character can flip too).

Control experiments that produce the correct 420: removing e1, or shrinking e1 to 8 units (≤ 2× the remainder). The bug triggers exactly when earn_amount − remainder > remainder.

Self-contained repro script (writes the ODS + config, runs rp2_us, prints the result)

# /// script
# requires-python = ">=3.11,<3.13"
# dependencies = ["rp2", "dali-rp2"]   # dali only for its ODS template
# ///
import configparser, subprocess, sys, tempfile
from pathlib import Path
import dali, ezodf

work = Path(tempfile.mkdtemp(prefix="rp2-hifo-repro-"))
IN_COLS = ["unique_id", "timestamp", "asset", "exchange", "holder", "transaction_type",
           "spot_price", "crypto_in", "crypto_fee", "fiat_in_no_fee", "fiat_in_with_fee",
           "fiat_fee", "notes"]
OUT_COLS = ["unique_id", "timestamp", "asset", "exchange", "holder", "transaction_type",
            "spot_price", "crypto_out_no_fee", "crypto_fee", "crypto_out_with_fee",
            "fiat_out_no_fee", "fiat_fee", "notes"]
ins = [
    ["b1", "2021-01-01 00:00:00+00:00", "TST", "exchange1", "alice", "BUY", 30, 10, None, 300, 300, 0, ""],
    ["b2", "2021-02-01 00:00:00+00:00", "TST", "exchange1", "alice", "BUY", 20, 10, None, 200, 200, 0, ""],
    ["b3", "2021-03-01 00:00:00+00:00", "TST", "exchange1", "alice", "BUY", 10, 10, None, 100, 100, 0, ""],
    ["e1", "2024-01-01 00:00:00+00:00", "TST", "exchange1", "alice", "STAKING", 5, 15, None, 75, 75, 0, ""],
]
outs = [
    ["s1", "2022-06-01 00:00:00+00:00", "TST", "exchange1", "alice", "SELL", 50, 4, 0, 4, 200, None, ""],
    ["s2", "2024-06-01 00:00:00+00:00", "TST", "exchange1", "alice", "SELL", 50, 20, 0, 20, 1000, None, ""],
]

ods_path = work / "input.ods"
template = Path(dali.__file__).resolve().parent / "data" / "template.ods"
doc = ezodf.newdoc("ods", str(ods_path), template=str(template))
for index in reversed([i for i, name in enumerate(doc.sheets.names()) if name.startswith("__")]):
    del doc.sheets[index]
sheet = ezodf.Sheet("TST", size=(40, 20))
doc.sheets += sheet
r = 0
for table, cols, rows in (("IN", IN_COLS, ins), ("OUT", OUT_COLS, outs)):
    sheet[r, 0].set_value(table); r += 1
    for c, name in enumerate(cols):
        sheet[r, c].set_value(name)
    r += 1
    for row in rows:
        for c, value in enumerate(row):
            if value is not None:
                sheet[r, c].set_value(value)
        r += 1
    sheet[r, 0].set_value("TABLE END"); r += 2
doc.save()

config = configparser.ConfigParser()
config.optionxform = str
config["general"] = {"assets": "TST", "exchanges": "exchange1", "holders": "alice"}
config["in_header"] = {n: str(i) for i, n in enumerate(IN_COLS)}
config["out_header"] = {n: str(i) for i, n in enumerate(OUT_COLS)}
config["intra_header"] = {n: str(i) for i, n in enumerate(
    ["unique_id", "timestamp", "asset", "from_exchange", "from_holder", "to_exchange",
     "to_holder", "spot_price", "crypto_sent", "crypto_received", "notes"])}
ini_path = work / "config.ini"
with open(ini_path, "w") as fh:
    config.write(fh)

result = subprocess.run(["rp2_us", "-n", "-o", str(work), str(ini_path), str(ods_path), "-m", "hifo"],
                        capture_output=True, text=True)
if result.returncode:
    sys.exit(result.stderr[-2000:])

full = ezodf.opendoc(str(work / "hifo_rp2_full_report.ods"))
for s in full.sheets:
    if s.name != "TST Tax":
        continue
    for row in s.rows():
        vals = [str(c.value) if c.value is not None else "" for c in row[:8]]
        if "SELL" in vals and vals[0] == "2024.0":
            basis = float(vals[7])
            print(f"2024 SELL cost basis: {basis}  ({'BUG (expected 420.0)' if basis != 420.0 else 'correct'})")

Root cause

Two pieces interact:

1. tax_engine.py pairs earn-typed events with an acquired lot before discarding it. In _create_unfiltered_taxable_event_set's pairing loop, the earn branch (tax_engine.py:130-146) correctly emits a lot-less GainLoss, but it reaches that branch only after get_next_taxable_event_and_amount / get_acquired_lot_for_taxable_event has already run a seek against the earn event:

# accounting_engine.py:179-184 — new (earn) event is newer, current lot has remainder
if acquired_lot:
    self._set_partial_amount(acquired_lot, new_acquired_lot_amount)   # remainder recorded ✓
(_, new_acquired_lot, _, new_acquired_lot_amount) = self.get_acquired_lot_for_taxable_event(
    new_taxable_event, acquired_lot, new_taxable_event_amount, new_acquired_lot_amount
)   # seeks a lot for the EARN event, with amount = earn_amount − remainder

(The comment on line 177 even says "and it's not earn-typed", but the code does not check it.)

2. The feature-based seek pops the heap and re-pushes conditionally (abstract_accounting_method.py, AbstractFeatureBasedAccountingMethod.seek_non_exhausted_acquired_lot):

lot_candidates.clear_partial_amount(selected_acquired_lot)
if selected_acquired_lot_amount > taxable_event_amount:
    self.add_selected_lot_to_heap(lot_candidates.acquired_lot_heap, selected_acquired_lot)

The pop-and-conditionally-re-push assumes the returned lot will actually be consumed by the taxable event. For an earn event nothing is consumed, so when remainder ≤ earn_amount − remainder the lot is popped, not re-pushed, and never seen by the heap again. The subsequent _set_partial_amount(lot, remainder) (accounting_engine.py:181, on the next event transition) faithfully records the remainder in the dict — but for feature-based methods the dict is only consulted for lots that come off the heap, so the lot is unreachable.

The chronological methods (FIFO/LIFO) are immune: their iterator walks the list by index and never loses candidates.

Real-world impact (how we found it)

Found while cross-checking RP2 against an independent lot-level HIFO implementation on a real multi-year portfolio: the asset's highest-basis lot was partially consumed by an early sale (both implementations agree on that sale), then a staking auto-compound (earn) larger than twice the remainder followed, and every later RP2 sale skipped the remainder — consuming visibly cheaper lots while the highest-basis lot stayed in open positions. Effect on that portfolio: capital losses understated by several hundred dollars across two tax years, plus part of one sale's proceeds flipped from long-term to short-term. Manual lot-by-lot reconstruction from RP2's own full report confirms the selection; the numbers tie to the cent under the corrected selection. Portfolios that stake the same asset they trade — a very common pattern — will hit the earn > 2 × remainder trigger routinely.

Notes

• Incorrect lot exhaustion in HIFO #73 reported the same symptom in 2022 against the pre-heap _seek_first_non_exhausted_acquired_lot; this is a regression of that fix under the heap architecture.
• HIFO and LOFO should use fiat with fee for lot selection and not spot price #150 (use basis-with-fee instead of spot for the sort key) is independent; fixing it would not fix this.
• Possible fix directions: skip lot-pairing entirely for earn-typed taxable events in get_next_taxable_event_and_amount (matching the intent of the accounting_engine.py:177 comment), or make the feature-based seek re-push unconditional and rely on the partial-amount dict (popped-and-exhausted lots already continue correctly).

[macanudo527]

@ahalekelly
Can you ask fable to run a merit/demerit analysis on the two possible fixes. What are the ramifactions and double check for possible flaws and bugs that might come up. ie. ask it to check its work.

[jayr0d]

@ahalekelly Thanks for the reproduction description, I can follow that clearly. Can you try reproducing this manually to confirm? If you're open to it, a pull request with updated unit tests would be incredibly helpful to get this resolved.

[ahalekelly]

Sorry I don’t have any more details than what I posted above, you guys are probably better equipped to assess this.

I actually had Fable build a whole replacement tax software for because of the IRS 2025 per-wallet tracking requirement that rp2 doesn’t support, and this bug surfaced when it was validating the other software against rp2. So this bug isn’t an issue for me personally, I just wanted to report it for you guys.