beancount_import matching broken for `securities_and_cash` ofx account? about beancount-import HOT 5 OPEN

I should mention: I've been using beancount-import on my bank accounts without trouble for a while now. Matching seems to work fine.

from beancount-import.

I'm also experiencing what I believe you're describing. When I import transactions that look like

2021-01-01 * "TRANSFER - PRETAX - Investment Expense"
  Assets:Vanguard:401k:PreTax:VGI007        -0.01 VGI007 {} @ 100.00 USD
    date: 2021-01-01
    ofx_fitid: "941930491093409139410394f"
    ofx_memo: "Investment Expense"
    ofx_type: "TRANSFER"
  Income:CapitalGains:Vanguard:401k:VGI007
  Expenses:Fees:Vanguard:401k                    1.00 USD

from the OFX importer and restart beancount-import, the same duplicate transaction is produced again, with an identical ofx_ftid, memo and type.

from beancount-import.

I don't think pull request #113 addresses this same issue.

The issue here is due to how matching.py (independent of OFX) works: currently matching is done based on posting weight, but as matching of the pending transaction is done without booking, the weight is unknown for these two postings:

  Assets:Brokerage:BMO         -13 BMO {} @ 76.03 USD
  Income:Capital-gains:BMO

Consequently, those postings don't participate in matching at all at the moment. Additionally, the remaining postings:

  Assets:Brokerage:Cash     988.37 USD
  Expenses:Fees               0.02 USD

don't balance (since we are missing the weight from the unknown-weight postings), and matching currently attempts to find a set of match groups that balance each currency.

To make this work the following improvement would be needed:

• If there are unknown weight postings in either transaction, we need to attempt an alternative matching scheme for the unknown weight postings. For example, we might allow an unknown-weight posting to match exactly one posting in the other transaction, and would allow for example the Assets:Brokerage:BMO posting to match based on the units and price alone, and the Income:Capital-gains:BMO posting would be allowed to match any posting with an account of Income:Capital-gains:BMO or Expenses:FIXME.

I'm looking into implementing that.

from beancount-import.

I pushed out a commit that solves part of the problem here.

However, it still does not actually make this case work. The problem is that matching is done against "partially booked" transactions, which still don't include resolved costs. Using fully-booked transactions is problematic because then the posting might actually have multiple lots and we will end up with multiple postings. Instead, I think the solution is to modify journal_editor to store the weight of each posting in the journal, which is already computed by booking, I believe. Then we can use that weight in matching.py, rather than deriving the weight from the posting itself (and not being able to determine the weight if the cost is not explicit).

from beancount-import.

I've dereferenced PR #126 from this issue. Thanks for the explanations and I'll have to read more to better understand how matching works. Hopefully PR #126 is still work submitting on its own.

from beancount-import.