Filing account when filed via bean-file is Assets:Investments:Taxable:Broker:{ticker}.

In 7e5caf3, this change was made:

When bean-file is called, filing_account has not been set (since initialize() is not run on a bean-file invoke), and main_account is used, which results in: Assets:Investments:Taxable:Broker:{ticker}.

See #55

I'd like to clean up the situation around payee/narration for good and do the right thing, whatever that is. It's fine to backward compatibility if needed for the benefit of getting this right for the longer run.

If at all you have the time and inclination figure out the full picture and submit a comprehensive solution, that would be great, and I'd rather accept that over the temporary fix of documenting it. If you're not inclined to, that's fine too: this needs to be documented and worked on at some point.

With that said, the things that come to mind are:

• Beancount reference
• solving for the comment here
• solving for the investments transaction builder
• ensuring the current importers don't break (it's okay if they don't do the same thing as earlier, and do the right thing instead; however, they shouldn't crash)
• ensuring smart_importer works correctly with the solution
• documenting methods to fix backward compatibility for users

It looks like the payee and transaction type fields are switched up in banking.Importer:

I've noticed that this was a conscious choice by reviewing older commits. I think generally the merchant who accepted the card should be recorded as the payee even if in some OFX formatted files follow another convention.

In these problematic cases the cleanest solution would be to correct this issue in the OFX parsing phase instead of doing that in a later stage of the pipeline where other parsers (such as the CSV parser) could also pump data and could reasonably expect that the payee field will be recorded as the payee eventually.

In my case I wrote a CSV parser which puts the payee information in the payee field and the narration information in memo. The generated transactions contained no payee information at all (apparently the transaction type goes there by default) and the narration field contained the value of the payee field. This is very confusing.

I recommend making a change where the payee field ends up as the payee of the transaction and the contents of memo or narration as the narration. In my opinion the transaction type is less important than the contents of a transfer memo field for example.

How are you getting fidelity ofx exports? I only see CSV and QIF options.

Hi Red,

Really liking your Beancount importers, so thanks for publishing them.

I'm trying to use the Workday importer to process everything related to my paycheck in one go, but unfortunately I'm getting a blank result to bean-identify _config.py workday.xlsx.

My _config.py looks like this:

from unittest import skip
from beancount.ingest.importers.csv import Importer as CSVImporter, Col
from beancount_reds_importers import workday

paycheck_template = {

        'EE Taxes': {
            "Federal Withholding": "Expenses:US:Taxes:Federal"
        }

    }

workday = workday.Importer({'desc': "Work",
    'main_account': 'Income:US:Hooli:Gross-Income',
    'paycheck_template': paycheck_template,
    'currency': 'USD',
})

CONFIG = [
    workday
    ]

With a sub-table heading "EE Taxes", and "Federal Withholding" being a row within that table.

I'm sure I'm missing something obvious, but I've been going round in circles so thought I'd ask just in case there was something else going on.

Hi, using your vanguard ofx importer again (thanks!). Perhaps it may be an anomaly but noticed that vanguard has a possibility of generating a dividend reinvestment with a unitprice of 0, triggering the "Either price ({}) or cost ({}) must be specified".format(price_number, cost_number) in libtransactionbuilder/common.py . This is more of a heads-up as I'm not sure yet if it's better to have a special case around this or expect the user to make a minor edit to their ofx file to deal with this. This may just be a a one-off anomaly since I don't see it with other dividend reinvest transactions.

Sample of what it looks like:

<REINVEST><INVTRAN><FITID>999999999<DTTRADE>20210922160000.000[-5:EST]<DTSETTLE>20210622160000.000[-5:EST]<MEMO>DIVIDEND REINVEST</INVTRAN><SECID><UNIQUEID>999999999<UNIQUEIDTYPE>CUSIP</SECID><INCOMETYPE>DIV<TOTAL>-9.99<SUBACCTSEC>CASH<UNITS>9.99<UNITPRICE>0.0</REINVEST>

Transaction(meta={'filename':                                                   
'/home/savingsandloan/OfxDownload.qfx', 'lineno': 106},        
date=datetime.date(2021, 9, 22), flag='*', payee='DIVIDEND REINVEST',           
narration='[VMRXX] Vanguard Cash Reserves Federal Money Market Fund - Admiral   
Shares', tags=frozenset(), links=frozenset(), postings=[])

Things also appear this way on the transactions table on their website, and the csv output.

This code results in an AttributeError, because ot.unit_price is immutable.

            # annoyingly, vanguard reinvests have unit_price set to zero. so manually compute it
            if (hasattr(ot, 'security') and ot.security) and ot.units and not ot.unit_price:
                ot.unit_price = round(abs(ot.total) / ot.units, 4)
            common.create_simple_posting_with_cost(entry, main_acct, units, ticker, ot.unit_price,
                                                   self.currency, self.price_cost_both_zero_handler)

It's a simple fix, just set a local variable unit_price and pass that into the create_simple_posting_with_cost directly. I don't think ot.unit_price is used beyond that:

            unit_price = ot.unit_price
            # annoyingly, vanguard reinvests have unit_price set to zero. so manually compute it
            if (hasattr(ot, 'security') and ot.security) and ot.units and not ot.unit_price:
                unit_price = round(abs(ot.total) / ot.units, 4)
            common.create_simple_posting_with_cost(entry, main_acct, units, ticker, unit_price,
                                                   self.currency, self.price_cost_both_zero_handler)

This may be an edge case, but I'm trying to import my data from a Tiller CSV extract which contains transaction activity for all of my configured accounts (banks, brokerage firms, credit cards, etc.) I've seen you can identify the main_account and cash_account dynamically using the {ticker} and {currency} variables but in looking through the code, I only see those two variables being replaced in the account names. Is there a way to access all columns from the original CSV file so I can reference those in the config passed to the Importer initializer? Something like

    'main_account'   : 'Assets:{Institution}:{Account}:{ticker}',

BTW, I'm using beancount-reds-importers==0.6.0 installed with pip.

When extracting example files OfxDownload.qfx works fine:
bean-extract my.import -f my.beancount OfxDownload.qfx

result:

;; -*- mode: beancount -*-
**** K:\red\OfxDownload.qfx

2019-02-27 * "Sell mutual fund" "[VGTEST] Vanguard Test Fund"
  Assets:Investments:TradIRA:VGTEST   400.1234 VGTEST {200.00 USD}
  Assets:Investments:TradIRA:USD     -80024.68 USD

2019-02-28 * "Sell mutual fund" "[VGTEST] Vanguard Test Fund"
  todo: "TODO: this entry is incomplete until lots are selected (bean-doctor context <filename> <lineno>)"
  Assets:Investments:TradIRA:VGTEST  -12.1234 VGTEST {} @ 200.00 USD
  Income:Capital-Gains:TradIRA
  Assets:Investments:TradIRA:USD      2424.68 USD

2019-03-01 balance Assets:Investments:TradIRA:VGTEST               388.00 VGTEST

2019-03-01 balance Assets:Investments:TradIRA:USD                  6543.21 USD

2019-03-31 price VGTEST                             200.00 USD

but transactions.qfx fails:
bean-extract my-smart.import -f my.beancount transactions.qfx

result:

ERROR:root:Importer beancount_reds_importers.ally.Importer.extract() raised an unexpected error: 'Statement' object has no attribute 'balance'
ERROR:root:Traceback: Traceback (most recent call last):
  File "c:\users\btycoon\appdata\local\programs\python\python38\lib\site-packages\beancount\ingest\extract.py", line 186, in extract
    new_entries = extract_from_file(
  File "c:\users\btycoon\appdata\local\programs\python\python38\lib\site-packages\beancount\ingest\extract.py", line 69, in extract_from_file
    new_entries = importer.extract(file, **kwargs)
  File "c:\users\btycoon\appdata\local\programs\python\python38\lib\site-packages\smart_importer\hooks.py", line 41, in patched_extract_method
    imported_entries = unpatched_extract(file, existing_entries)
  File "c:\users\btycoon\appdata\local\programs\python\python38\lib\site-packages\beancount_reds_importers\libtransactionbuilder\banking.py", line 97, in extract
    new_entries += self.extract_balance(file, counter)
  File "c:\users\btycoon\appdata\local\programs\python\python38\lib\site-packages\beancount_reds_importers\libtransactionbuilder\banking.py", line 60, in extract_balance
    for bal in self.get_balance_statement():
  File "c:\users\btycoon\appdata\local\programs\python\python38\lib\site-packages\beancount_reds_importers\libreader\ofxreader.py", line 52, in get_balance_statement
    yield Balance(date, self.ofx_account.statement.balance)
AttributeError: 'Statement' object has no attribute 'balance'

;; -*- mode: beancount -*-

(i am using master, bean-extract works for both files)

Examples:
#5

https://groups.google.com/g/beancount/c/azhLV-Om7dU

The Fava API was having trouble jsonifying imports. Turns out it was because of a single count object that made its way into metadata for balance entries. I looked at the data it was attempting to run through json.dumps and saw this:

    transaction = {
        "meta": {
            "filename": "ChaseXXXX....QFX",
            "lineno": count(250),
        },
        "date": datetime.date(2023, 5, 5),
        "account": "Liabilities:Current-Liabilities:Credit-Cards:Chase:Sapphire-Reserve",
        "amount": {"number": "-5442.48", "currency": "USD"},
        "tolerance": None,
        "diff_amount": None,
        "type": "Balance",
    }

I suspect it's because next() isn't being used on this metadata initialization

Suspected fix is to do next(counter).

edit: tested this fix on my machine--works!

Instead of opening an issue for each is it ok to list them in one place like this?

Well if this offends you or you'd like it another way I'll change for the next time. :)

1. two links in the README.md give 404 errors

"See working examples of an ofx based and csv based importers."

2. the example directory does not run (likely due to import and name changes).

#72 should have broken the schwab_csv_brokerage unit test, which it did not. Fix.

Running the latest importer version for scbbank. Ran into the error below when importing as usual this month. Same error for all csv files. Even re-tried importing old csv files which ran fine before but generate this error now.

ERROR:root:Importer beancount_reds_importers.importers.stanchart.scbbank.Importer.extract() raised an unexpected error: '<' not supported between instances of 'method' and 'datetime.date'

When I run my-smart.import with bean-extract and no pre-existing data everything runs fine.

Once I add in actual data, I get the error:
ERROR:root:Importer beancount_reds_importers.chase.Importer.extract() raised an unexpected error: transform() takes 1 positional argument but 2 were given

I found this:
https://stackoverflow.com/questions/23944657/typeerror-method-takes-1-positional-argument-but-2-were-given

And I was able to go to smart_importer/pipelines.py and and add self to transform() like this:
def transform(self, data):

After a very quick check, it looks to have resolved my issue. Checking if this makes sense and if there might be an actual bug

unexpected error: 'AmexCCImporter' object has no attribute 'file_read_done'

Fix might be to add 'self.initialize_reader(file)' at the beginning of this method, but I want to figure out what caused this.

From here.

Hello,

I ran into an issue trying to parse downloads from Fidelity accounts that have banking features enabled (in particular accounts that receive paper checks and use bill payment...I could see there being similar issues with writing checks, but I don't have any examples to try).

The OFX file looks something like this for the transactions that are failing:

<INVBANKTRAN>    <STMTTRN>
	<TRNTYPE>DEP</TRNTYPE>
<DTPOSTED>20220909000000.000[-5:EDT]</DTPOSTED>
<TRNAMT>+00000000000001.0000</TRNAMT>
<FITID>XXXXXXXXXXXXX1220220909</FITID>
<NAME>CHECK RECEIVED</NAME>
<MEMO>CHECK RECEIVED</MEMO>
<CURRENCY>
	<CURRATE>1.00</CURRATE>
<CURSYM>USD</CURSYM>
</CURRENCY>    </STMTTRN>
<SUBACCTFUND>CASH</SUBACCTFUND>    </INVBANKTRAN>
<INVBANKTRAN>    <STMTTRN>
	<TRNTYPE>PAYMENT</TRNTYPE>
<DTPOSTED>20220909000000.000[-5:EDT]</DTPOSTED>
<TRNAMT>-00000000000001.0000</TRNAMT>
<FITID>XXXXXXXXXXXXX1120220909</FITID>
<NAME>BILL PAYMENT         MYPAYEE NA</NAME>
<MEMO>BILL PAYMENT         MYPAYEE NAME            /0047/Y*******</MEMO>
<CURRENCY>
	<CURRATE>1.00</CURRATE>
<CURSYM>USD</CURSYM>
</CURRENCY>    </STMTTRN>
<SUBACCTFUND>CASH</SUBACCTFUND>    </INVBANKTRAN>

and the exceptions are e.g.

ERROR: unknown entry type: payment

However, the fix is quite simple, extending the cash transaction lists on line 257 and 318 to include:

elif ot.type in ['other', 'credit', 'debit', 'dep', 'cash', 'payment', 'check']:

elif ot.type in ['other', 'credit', 'debit', 'transfer', 'dep', 'income',
                             'dividends', 'capgainsd_st', 'capgainsd_lt', 'cash', 'payment', 'check']:

Allows these to be processed like other cash transactions.

When doing my last update, I noticed the header for Schwab's checking CSV has changed, which breaks the importer.

I just modified the header and CSV file in order to use the current importer, but going forward the Checking CSV importer should change.

The old header:

"Transactions  for Checking account ...XXX as of 09/02/2023 01:46:01 PM ET"
"Date","Type","Check #","Description","Withdrawal (-)","Deposit (+)","RunningBalance"
"Pending Transactions are not reflected within this sort criterion."
"Posted Transactions"

The new header:

"Date","Status","Type","CheckNumber","Description","Withdrawal","Deposit","RunningBalance"

I can make the changes to the importer the next time I update my ledger and put in a PR, but also flagging this here in case others run into the same issue.

Ran into the following issue while trying to use the import page in Fava. Error comes from their API:

{
    "error": "Error calling importer method: 'NoneType' object has no attribute 'statement'",
    "success": false
}

Tracked the error down with PDB to the line

> /Users/me/Library/Mobile Documents/com~apple~CloudDocs/Personal/Code/finances/venv/lib/python3.9/site-packages/beancount_reds_importers/libreader/ofxreader.py(41) 
file_date() -> return self.ofx_account.statement.end_date
(Pdb) list
 36             the full account number. Override this method to handle these cases."""
 37             return file_account == config_account
 38  
 39         def file_date(self, file):
 40             "Get the maximum date from the file."
 41  ->         return self.ofx_account.statement.end_date
 42  

Finding this in source identifies that it is never checked if ofx_account is None after the logic in the __init__.

I suspect this is a quirk of Chase QFX files. Perhaps there is a better way of getting the latest date in the file?

Is there somewhere an option to define column names for csv files that don't have a header?

When running an import on a vanguard ofx document that has CDs in it I get

ERROR:root:Importer beancount_reds_importers.vanguard.Importer.extract() raised an unexpected error: Unknown entry type
Traceback (most recent call last):
File "/Users/kaseyklipsch/pf/.venv/lib/python3.11/site-packages/beancount/ingest/extract.py", line 182, in extract
new_entries = extract_from_file(
^^^^^^^^^^^^^^^^^^
File "/Users/kaseyklipsch/pf/.venv/lib/python3.11/site-packages/beancount/ingest/extract.py", line 67, in extract_from_file
new_entries = importer.extract(file, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kaseyklipsch/pf/.venv/lib/python3.11/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 423, in extract
new_entries += self.extract_transactions(file, counter)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kaseyklipsch/pf/.venv/lib/python3.11/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 351, in extract_transactions
raise Exception('Unknown entry type')

An example line:
20688946420230613160000.000[-5:EST]20230615160000.000[-5:EST]BUY02007GP98CUSIP20000.0100.0-20000.0CASHCASH0.0

I received an OFX file from E*Trade that contains an unexpected entity, and this breaks the beancount_reds_importers code like so:

ERROR: unknown entry type: jrnlsec
ERROR:root:Importer beancount_reds_importers.etrade.Importer.extract() raised an unexpected error: Unknown entry type
Traceback (most recent call last):
File "/Users/chris/venv/lib/python3.11/site-packages/beancount/ingest/extract.py", line 182, in extract
new_entries = extract_from_file(
^^^^^^^^^^^^^^^^^^
File "/Users/chris/venv/lib/python3.11/site-packages/beancount/ingest/extract.py", line 67, in extract_from_file
new_entries = importer.extract(file, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/venv/lib/python3.11/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 422, in extract
new_entries += self.extract_transactions(file, counter)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/venv/lib/python3.11/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 350, in extract_transactions
raise Exception('Unknown entry type')
Exception: Unknown entry type
;; -- mode: beancount --

FWIW, I just added 'jrnlsec' to the list of entities on line 146 of investments.py

The line appeared in my OFX after an incoming ACATS transfer and it looks like this:
<JRNLSEC> <INVTRAN> <FITID>230614_JRL_0 <DTTRADE>20230614170000.000 <MEMO>Journalling security holdings for VTEB </INVTRAN> <SECID> <UNIQUEID>922907746 <UNIQUEIDTYPE>CUSIP </SECID> <SUBACCTTO>CASH <SUBACCTFROM>MARGIN <UNITS>100 </JRNLSEC>

I'm very new to Beancount and tried running the examples from your Readme, but run into the following errors:

> ./import.sh OfxDownload.qfx
**** /home/bart/.bart-beancount/beancount-env/lib/python3.9/site-packages/beancount_reds_importers/example/OfxDownload.qfx
Importer:    beancount_reds_importers.vanguard.Importer
Account:     Assets:Investments:TradIRA

ERROR:root:Importer beancount_reds_importers.vanguard.Importer.extract() raised an unexpected error: 'interest'
Traceback (most recent call last):
  File "/home/bart/.bart-beancount/beancount-env/lib/python3.9/site-packages/beancount/ingest/extract.py", line 182, in extract
    new_entries = extract_from_file(
  File "/home/bart/.bart-beancount/beancount-env/lib/python3.9/site-packages/beancount/ingest/extract.py", line 67, in extract_from_file
    new_entries = importer.extract(file, **kwargs)
  File "/home/bart/.bart-beancount/beancount-env/lib/python3.9/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 306, in extract
    self.initialize(file)
  File "/home/bart/.bart-beancount/beancount-env/lib/python3.9/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 45, in initialize
    self.build_account_map()  # TODO: avoid for identify()
  File "/home/bart/.bart-beancount/beancount-env/lib/python3.9/site-packages/beancount_reds_importers/libtransactionbuilder/investments.py", line 57, in build_account_map
    "income":    self.config['interest'],
KeyError: 'interest'
;; -*- mode: beancount -*-

 ./import.sh transactions.qfx
usage: bean-identify [-h] [--version] CONFIG_FILENAME DIR-OR-FILE [DIR-OR-FILE ...]
bean-identify: error: File does not exist: 'transactions.qfx'
usage: bean-extract [-h] [--version] [-e BEANCOUNT_FILE] [-r] CONFIG_FILENAME DIR-OR-FILE [DIR-OR-FILE ...]
bean-extract: error: File does not exist: 'transactions.qfx'

I notice that there is a transactions.qfx file in your master branch. Is the Python package out of date, or am I doing something wrong?

hello, this is "Fin" from the beancount forum.

I've hit this error message even when I have:

    self.include_balances = False

defined in my custom init for the csv converter.

To temporarily get around it I've changed the raise error to be commented out and returned (()) instead. If I simply say pass it will error because it wanted something back.

My data file that I'm testing at the moment has no balance assertions or any positions as it is only a small part I'm working on for testing my tdameritrade csv importer.

I'm sure I could also be doing something else wrong, but at least with this fix I can get output for where I am at for now. :)

With the latest addition of bean-download and the from click_aliases import ClickAliasedGroup in util/bean_ingest.py, does click_aliases need to be added as a dependency here?
https://github.com/redstreet/beancount_reds_importers/blob/main/requirements.txt

Updated to the latest version from git (not sure if this is the real issue) and ran into the following errors when importing:

`Traceback (most recent call last):
File "/[HOME_PATH]/.local/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 126, in feed_token
action, arg = states[state][token.type]
~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'EOL'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/[HOME_PATH]/.local/bin/autobean-format", line 8, in
sys.exit(main())
^^^^^^
File "/[HOME_PATH]/.local/lib/python3.11/site-packages/autobean_format/main.py", line 83, in main
for file in formatter.load_files(filename):
File "/[HOME_PATH]/.local/lib/python3.11/site-packages/autobean_format/main.py", line 49, in load_files
model = self._parser.parse(text, models.File)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/[HOME_PATH]/.local/lib/python3.11/site-packages/autobean_refactor/parser.py", line 120, in parse
model = self._parse(text, target, self._lark)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/[HOME_PATH]/.local/lib/python3.11/site-packages/autobean_refactor/parser.py", line 131, in _parse
parser.feed_token(token)
File "/[HOME_PATH]/.local/lib/python3.11/site-packages/lark/parsers/lalr_interactive_parser.py", line 32, in feed_token
return self.parser_state.feed_token(token, token.type == '$END')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/[HOME_PATH]/.local/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 129, in feed_token
raise UnexpectedToken(token, expected, state=self, interactive_parser=None)
lark.exceptions.UnexpectedToken: Unexpected token Token('EOL', '') at line 71, column 71.
Expected one of:
* TILDE
* CURRENCY

`

How can I help diagnose this? Thanks!

I'm basing my importer on schwab_csv_brokerage.py. There is no suitable type field that can be mapped. Instead there is a DESCRIPTION column with values such as "Sold 30 BP @ 28.15". Is there a way to map types with transaction_type_map and regexps or lambdas?

Here is the CSV:

DATE,TRANSACTION ID,DESCRIPTION,QUANTITY,SYMBOL,PRICE,COMMISSION,AMOUNT,REG FEE,SHORT-TERM RDM FEE,FUND REDEMPTION FEE, DEFERRED SALES CHARGE
06/10/2021,35392489790,CLIENT REQUESTED ELECTRONIC FUNDING RECEIPT (FUNDS NOW),,,,,6000.00,,,,
06/11/2021,35417292898,Bought 30 BP @ 27.71,30,BP,27.71,0.00,-831.30,,,,
06/11/2021,35417258165,Bought 50 LUV @ 57.44,50,LUV,57.44,0.00,-2872.00,,,,
06/11/2021,35417318519,Bought 5 COST @ 383.86,5,COST,383.86,0.00,-1919.30,,,,
08/13/2021,36765798623,QUALIFIED DIVIDEND (COST),,COST,,,3.95,,,,
09/24/2021,37594162919,QUALIFIED DIVIDEND (BP),,BP,,,9.83,,,,
09/24/2021,37594162920,ADR FEE (BP),,BP,,,-0.15,,,,
09/27/2021,37623582592,Bought 7 ON @ 49.55,7,ON,49.55,0.00,-346.85,,,,
10/25/2021,38216559411,Sold 5 COST @ 490.045,5,COST,490.045,0.00,2450.22,0.01,,,
11/03/2021,38462866073,Sold 7 ON @ 57.85,7,ON,57.85,0.00,404.95,,,,
11/05/2021,38535502875,Bought 38 VLO @ 76.28,38,VLO,76.28,0.00,-2898.64,,,,
12/09/2021,39262753915,QUALIFIED DIVIDEND (VLO),,VLO,,,37.24,,,,
12/17/2021,39494229796,QUALIFIED DIVIDEND (BP),,BP,,,9.68,,,,
12/23/2021,39634430392,QUALIFIED DIVIDEND (BP),,BP,,,0.15,,,,
12/23/2021,39634793012,QUALIFIED DIVIDEND (BP),,BP,,,-0.15,,,,
***END OF FILE***

My importer:

""" TD Ameritrade .csv importer."""

from beancount_reds_importers.libreader import csvreader
from beancount_reds_importers.libtransactionbuilder import investments


class Importer(csvreader.Importer, investments.Importer):
    IMPORTER_NAME = 'TD Ameritrade CSV'

    def custom_init(self):
        self.max_rounding_error = 0.04
        self.filename_pattern_def = 'transactions'
        self.header_identifier = 'DATE,TRANSACTION ID,DESCRIPTION.*'
        self.get_ticker_info = self.get_ticker_info_from_id
        self.date_format = '%m/%d/%Y'
        self.funds_db_txt = 'funds_by_ticker'
        self.skip_head_rows = 0
        self.skip_tail_rows = 1
        self.header_map = {
            "DATE":        'date',
            "TRANSACTION ID": 'memo',
            "DESCRIPTION": 'type',
            "SYMBOL":      'security',
            "QUANTITY":    'units',
            "PRICE":       'unit_price',
            "AMOUNT":      'amount',
            "SHORT-TERM RDM FEE": 'short_term_fee',
            "FUND REDEMPTION FEE": 'fund_redmpt_fee',
            " DEFERRED SALES CHARGE": 'deferred_sales_charge',
            "REG FEE": 'fees',
        }
        self.transaction_type_map = {}
        self.skip_transaction_types = []

    def get_target_acct_custom(self, transaction, ticker=None):
        if transaction.starts_with("Bought"):
            return self.config['buy']
        if transaction.starts_with("Sold"):
            return self.config['sell']
        return None

Many thanks @redstreet for making these importer tools available.

I tried to run the example importer in beancount_reds_importers/example (in a fresh virtual environment) but got the following error.

$ bean-identify my.import OfxDownload.qfx 
[...]
ModuleNotFoundError: No module named 'petl'

Looks like petl needs to be added as a dependency in setup.py?

Income type fields seem to be used inconsistently by institutions. We can still try to guess where possible.

From #41 (comment)

With a version of investments.py that includes the changes discussed here. I also very lightly modified the provided fidelity class to look like this:

class Importer(investments.Importer, ofxreader.Importer):
    IMPORTER_NAME = 'Fidelity Net Benefits / Fidelity Investments OFX'

    def custom_init(self):
        self.max_rounding_error = 0.14
        self.filename_pattern_def = '.*fidelity'
        self.get_ticker_info = self.get_ticker_info_from_id
        self.get_payee = lambda ot: ot.memo.split(";", 1)[0] if ';' in ot.memo else ot.memo

    def security_narration(self, ot):
        ticker, ticker_long_name = self.get_ticker_info(ot.security)
        return f"[{ticker}]"

    def file_name(self, file):
        return 'fidelity-{}-{}'.format(self.config['account_number'], ntpath.basename(file.name))

    def get_target_acct_custom(self, transaction, ticker=None):
        if transaction.memo.startswith("CONTRIBUTION"):
            return self.config['transfer']
        if transaction.memo.startswith("FEES"):
            return self.config['fees']
        elif transaction.type == 'income' and getattr(transaction, 'income_type', None) == 'DIV':
            return self.target_account_map.get('dividends', None)
        elif getattr(transaction, 'income_type', None) == "CGLONG":
            return self.target_account_map.get('capgainsd_lt', None)
        elif getattr(transaction, 'income_type', None) == "CGSHORT":
            return self.target_account_map.get('capgainsd_st', None)
        else:
            return None

The issue I ran into was that CGLONG and CGSHORT distributions were showing up in the OFX with a transaction.type of income, and then would be classified as dividends based on this code in investments.py:

def get_target_acct(self, transaction, ticker):
        target = self.get_target_acct_custom(transaction, ticker)
        if target:
            return target
        if transaction.type == 'income' and getattr(transaction, 'income_type', None) == 'DIV':
            return self.target_account_map.get('dividends', None)
        return self.target_account_map.get(transaction.type, None)

Wanted to put up an example of a Schwab banking CSV, as it seems to error out with the current format. Stack trace:

(.venv) doughepi@Pipers-MacBook-Pro finances %  bean-extract config.py ./data
ERROR:root:Importer beancount_reds_importers.importers.schwab.schwab_csv_checking.Importer.extract() raised an unexpected error: selection is not a field or valid field index: 'Date'
Traceback (most recent call last):
  File "/Users/doughepi/Library/Mobile Documents/com~apple~CloudDocs/Personal/Code/finances/.venv/lib/python3.9/site-packages/beancount/ingest/extract.py", line 182, in extract
    new_entries = extract_from_file(
  File "/Users/doughepi/Library/Mobile Documents/com~apple~CloudDocs/Personal/Code/finances/.venv/lib/python3.9/site-packages/beancount/ingest/extract.py", line 67, in extract_from_file
    new_entries = importer.extract(file, **kwargs)
  File "/Users/doughepi/Library/Mobile Documents/com~apple~CloudDocs/Personal/Code/finances/.venv/lib/python3.9/site-packages/smart_importer/hooks.py", line 40, in patched_extract_method
    imported_entries = unpatched_extract(
  File "/Users/doughepi/Library/Mobile Documents/com~apple~CloudDocs/Personal/Code/finances/.venv/lib/python3.9/site-packages/beancount_reds_importers/libtransactionbuilder/banking.py", line 107, in extract
    self.read_file(file)
  File "/Users/doughepi/Library/Mobile Documents/com~apple~CloudDocs/Personal/Code/finances/.venv/lib/python3.9/site-packages/beancount_reds_importers/libreader/csvreader.py", line 180, in read_file
    rdr = self.convert_columns(rdr)
  File "/Users/doughepi/Library/Mobile Documents/com~apple~CloudDocs/Personal/Code/finances/.venv/lib/python3.9/site-packages/beancount_reds_importers/libreader/csvreader.py", line 99, in convert_columns
    if 'type' in rdr.header():
  File "/Users/doughepi/Library/Mobile Documents/com~apple~CloudDocs/Personal/Code/finances/.venv/lib/python3.9/site-packages/petl/util/base.py", line 338, in header
    return tuple(next(it))
  File "/Users/doughepi/Library/Mobile Documents/com~apple~CloudDocs/Personal/Code/finances/.venv/lib/python3.9/site-packages/petl/transform/headers.py", line 90, in iterrename
    raise FieldSelectionError(x)
petl.errors.FieldSelectionError: selection is not a field or valid field index: 'Date'
;; -*- mode: beancount -*-

Here's an example of the CSV:

"Transactions  for Checking account ...000 as of 05/10/2023 01:41:36 PM ET"
"Date","Type","Check #","Description","Withdrawal (-)","Deposit (+)","RunningBalance"
"Pending Transactions"
"Total Pending Check and other Credit(s)","","","","","$0.00",""
"05/09/2023","","","APPLE CASH CUPERTINO, CA, US","$130.00","",""
"05/09/2023","","","APPLE CASH CUPERTINO, CA, US","$50.00","",""
"05/09/2023","","","APPLE CASH CUPERTINO, CA, US","$15.00","",""
"Posted Transactions"
"12/30/2022","INTADJUST","","Interest Paid","","$1.00","$3,951.00"
"12/25/2022","VISA","","Grocery Store","$50.00","","$3,950.00"
"12/20/2022","TRANSFER","","Funds Transfer to Brokerage -XXXX","$1,000.00","","$4,000.00"
"12/14/2022","ACH","","Electronic Deposit","","$5,000.00","$5,000.00"

From #70

1. "Cash Withdrawal" entries should have 2 decimal places even for whole numbers but are currently missing. This affects automatic numerical display inference in Fava

Temporarily fixed. Still needs a full solution. The problem is, petl uses xlwt/xlrd to read excel, which use excel datatypes, which we don't want, as we want just the string, which we can then pass on to decimal.Decimal. Not sure exactly how decimals are stored in excel

When I run pytest --generate I get an error that xlrd isn't installed. Installing this manually with pip install xlrd fixes the problem but it would be great if that dependency were automatically installed.

Unfortunately I'm not savvy enough with Python to know how to fix this- I spent some time searching but I wasn't able to find a solution to this.

self = <uobbank_test.TestUOB object at 0x1030d7010>, importer = <beancount_reds_importers.importers.unitedoverseas.uobbank.Importer object at 0x1030d4f50>
file = <beancount.ingest.cache._FileMemo object at 0x1030d5dd0>

    def test_identify(self, importer, file):
        """Attempt to identify a file and expect results to be true.
    
        This method does not need to check against an existing expect file. It
        is just assumed it should return True if your test is setup well (the
        importer should always identify the test file).
        """
>       assert importer.identify(file)

../../.asdf/installs/python/3.11.4/lib/python3.11/site-packages/beancount/ingest/regression_pytest.py:123: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
beancount_reds_importers/libreader/reader.py:26: in identify
    self.initialize_reader(file)
beancount_reds_importers/libreader/xlsreader.py:22: in initialize_reader
    for r in rdr:
../../.asdf/installs/python/3.11.4/lib/python3.11/site-packages/petl/io/xls.py:43: in __iter__
    wb = xlutils_view.View(source3, **self.kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <petl.io.xlutils_view.View object at 0x103124c90>
file_contents = b'\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x03\x00\xfe\xf...0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00(\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00'
class_ = None, kwargs = {'logfile': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>}

    def __init__(self, file_contents, class_=None, **kwargs):
        self.class_ = class_ or self.class_
>       from xlrd import open_workbook
E       ModuleNotFoundError: No module named 'xlrd'

../../.asdf/installs/python/3.11.4/lib/python3.11/site-packages/petl/io/xlutils_view.py:116: ModuleNotFoundError

The proposed changes would add support for Commodity & Account entries

• Readers: Add get_commodity_entries and get_account_entries methods to readers (ofxreader, csvreader, csv_multitable_reader)
• Common Importers: Add configurable checks of self.include_commodities and self.include_accounts to investments.py (any other relevant importers?)
• Common Importers: Add extract_commodities and extract_accounts methods that would be called within extract if associated include_ is set to true.

An earlier approach to this in fidelity_csv_old was commits:

• savingsandloan@c17331f
• savingsandloan@a713313

It would largely be the same except of course carry the new approach to build_metadata as done in #28

My csv file has the following Columns
Date,Transaction Title,Amount,Paid from Account,Credit to Account,Receipt

e.g.
20/04/2019,Fuel,1000,BankCard,Travel,https://drive.google.com/image_url

In my custom_init I am setting header_map to
'Date': 'date',
'Transaction Title': 'memo',
'Amount': 'Amount',

How should I map the remaining 3 columns?

I'm using Fidelity for most of my financial life. Part of that includes using their cash management account and a brokerage account that acts like a checking account.

I'm using the importers for the investment brokerage accounts just fine, but when I try to apply a banking.Importer for the more "bank"-like brokerage accounts I get the error that InvestmentStatement object has no attribute 'balance'. Traceback at the bottom.

I tried creating a custom importer for the "banking" accounts with self.include_balances = False to skip adding balances, but still get the same error.

class Importer(banking.Importer, ofxreader.Importer):
    IMPORTER_NAME = "Fidelity Banking Accounts"

    def custom_init(self):
        if not self.custom_init_run:
            self.max_rounding_error = 0.04
            self.filename_pattern_def = "fidelity_banking*"
            self.includes_balances = False
            self.custom_init_run = True

oft-summarize works, but seems to leave the balance as None:

Statement info: 2022-12-01 00:00:00 -- 2022-12-29 00:00:00. Bal: []

I don't mind not having balance info for these accounts. Would it be acceptable to try and warn for this behavior, or is there something likely wrong with OFX file?

ERROR:root:Importer personal_finance.ingestors.beancount_reds_importers.fidelity_banking.Importer.extract() raised an unexpected error: 'InvestmentStatement' object has no attribute 'balance'
Traceback (most recent call last):
  File "~/pypoetry/virtualenvs/personal-finance-Iy5H7pvA-py3.9/lib/python3.9/site-packages/beancount/ingest/extract.py", line 182, in extract
    new_entries = extract_from_file(
  File "~/pypoetry/virtualenvs/personal-finance-Iy5H7pvA-py3.9/lib/python3.9/site-packages/beancount/ingest/extract.py", line 67, in extract_from_file
    new_entries = importer.extract(file, **kwargs)
  File "~/pypoetry/virtualenvs/personal-finance-Iy5H7pvA-py3.9/lib/python3.9/site-packages/smart_importer/hooks.py", line 40, in patched_extract_method
    imported_entries = unpatched_extract(
  File "~/pypoetry/virtualenvs/personal-finance-Iy5H7pvA-py3.9/lib/python3.9/site-packages/beancount_reds_importers/libtransactionbuilder/banking.py", line 119, in extract
    new_entries += self.extract_balance(file, counter)
  File "~/pypoetry/virtualenvs/personal-finance-Iy5H7pvA-py3.9/lib/python3.9/site-packages/beancount_reds_importers/libtransactionbuilder/banking.py", line 67, in extract_balance
    for bal in self.get_balance_statement():
  File "~/pypoetry/virtualenvs/personal-finance-Iy5H7pvA-py3.9/lib/python3.9/site-packages/beancount_reds_importers/libreader/ofxreader.py", line 56, in get_balance_statement
    yield Balance(date, self.ofx_account.statement.balance)
AttributeError: 'InvestmentStatement' object has no attribute 'balance'

I've been noticing the vanguard ofx importer will create a duplicate balance assertion for the money market account.

Sometimes it may be the same amount as the money market account:

2021-12-01 balance Assets:Vanguard:TradIRA:USD           3.33 USD

2021-12-01 balance Assets:Vanguard:TradIRA:VMFXX         3.33 VMFXX

Sometimes I think it may just be zero, e.g.:

2021-12-01 balance Assets:Vanguard:TradIRA:USD           0.00 USD

2021-12-01 balance Assets:Vanguard:TradIRA:VMFXX         3.33 VMFXX

Using the Schwab Brokerage CSV importer with the following entry seemingly generates a $0 transaction. Seems like a bug.

"Date","Action","Symbol","Description","Quantity","Price","Fees & Comm","Amount"
"01/09/2023","MoneyLink Deposit","","My Name","","","","$25.00"

Results in

2022-12-27 * "transfer" "transfer"
  Assets:Long-Term-Investments:Charles-Schwab:General-Investing:USD  0 USD

Based on my config

CONFIG = [
    schwab_csv_brokerage.Importer(
        {
            "main_account": "Assets:Long-Term-Investments:Charles-Schwab:General-Investing:{ticker}",
            "cash_account": "Assets:Long-Term-Investments:Charles-Schwab:General-Investing:{currency}",
            "account_number": "XXX",
            "dividends": "Income:Dividends:Schwab:General-Investing:{ticker}",
            "interest": "Income:Interest:Schwab:General-Investing:{ticker}",
            "cg": "Income:Capital-Gains:Schwab:General-Investing:{ticker}",
            "capgainsd_lt": "Income:Capital-Gains-Distributions:Long:Schwab:General-Investing:{ticker}",
            "capgainsd_st": "Income:Capital-Gains-Distributions:Short:Schwab:General-Investing:{ticker}",
            "fees": "Expenses:Brokerage-Fees:Schwab",
            'rounding_error' : 'Equity:Rounding-Errors',
            "currency": "USD",
            "fund_info": fund_info,
        }
    ),
]

I think this has to do with https://github.com/redstreet/beancount_reds_importers/blob/main/beancount_reds_importers/libtransactionbuilder/investments.py#LL271C17-L277C17 which treats Schwab MoneyLink Deposit and MoneyLink Transfer actions as reds_importer transfers, but they lack a unit. It should be using the total in this case.

Let me know! I'm not sure what the best fix would be, as I'm new to this library. I also noticed in the code that Journal actions are ignored. Should MoneyLink actions be ignored too? They're also just plain account deposits and withdrawals, just like Journal.

Hi all, I am (very) new to python and trying my best to get the importers up and running (Thanks Red!) but I am encountering some error messages which I unfortunately cannot understand. Some pointers or tips on where I went wrong will be very much appreciated!

From my rough understanding, it seems to have something to do with dates - but I do think I've set the date format correctly in the CONFIG file, and also double-checked that the date format in CSV aligns - it's DD-MM-YYYY in the raw csv file.

ERROR MESSAGE:

**** /mnt/tank/[XXX]/.local/lib/python3.10/site-packages/beancount_reds_importers/scb/AccountTransactions230115231624359204879.csv
Importer: beancount_reds_importers.scb.Importer
Account: Assets:Bank:SCB

ERROR:root:Importer beancount_reds_importers.scb.Importer.extract() raised an unexpected error: 'NoneType' object has no attribute 'date'
Traceback (most recent call last):
File "/mnt/tank/[XXX]/.local/lib/python3.10/site-packages/beancount/ingest/extract.py", line 182, in extract
new_entries = extract_from_file(
File "/mnt/tank/[XXX]/.local/lib/python3.10/site-packages/beancount/ingest/extract.py", line 67, in extract_from_file
new_entries = importer.extract(file, **kwargs)
File "/mnt/tank/[XXX]/.local/lib/python3.10/site-packages/beancount_reds_importers/libtransactionbuilder/banking.py", line 107, in extract
entry = data.Transaction(metadata, ot.date.date(), self.FLAG,
AttributeError: 'NoneType' object has no attribute 'date'
;; -- mode: beancount --

From MY.CONFIG

from beancount_reds_importers.libreader import csvreader
from beancount_reds_importers.libtransactionbuilder import banking

class Importer(banking.Importer, csvreader.Importer):
def custom_init(self):
self.max_rounding_error = 0.04
self.date_format = '%d-%m-%Y'

@patbakdev writes:

I have about 10 Fidelity Investment, NetBenefits, and CMA accounts. I am currently using OFXClient to download and red's fidelity OFX importer to handle all of them. Unfortunately one of them started to give me a download error and I think we are quickly approaching the end of simple CLI download of OFX data. It seems we will all have to manually login and download unless you are Intuit. Fortunately, Fidelity will allow me to download all of the account data in one file. The bad news is that they only offer CSV.

I'd love to have a single all-in-one Fidelity CSV importer that can handle such a file. I tried using the Fidelity CSA importer against it, but it didn't recognize it (I assume it doesn't support multiple accounts). Given the comments above, it seems like basing something off of the investment builder makes the most sense. Has anyone given any thought to such an importer. I have another project I am currently working, on, but I have a feeling I am going to have to deal with this sooner than later.

EDIT: I'm looking at the schwab importer now. Maybe we need somethings similar like fidelity_{csv, ofx}_{balances, checking, brokerage}.py

Do I have to do something special to get a CUSIP # from QOL?

I tried searching Vanguard's VTWAX and getting nothing. Also noticed that one of the CUSIP #'s in the example fund_info.py comes up not found on QOL. Am I doing it wrong?

installed via pip install.

any bean-download command fails in click's init call:

❯ bean-download config-template
Traceback (most recent call last):
  File "/home/huyang/.local/bin/bean-download", line 5, in <module>
    from beancount_reds_importers.util.bean_download import cli
  File "/home/huyang/.local/lib/python3.8/site-packages/beancount_reds_importers/util/bean_download.py", line 85, in <module>
    def download(config_file, sites, site_types, dry_run, verbose):  # noqa: C901
  File "/home/huyang/.local/lib/python3.8/site-packages/click/decorators.py", line 192, in decorator
    _param_memo(f, OptionClass(param_decls, **option_attrs))
  File "/home/huyang/.local/lib/python3.8/site-packages/click/core.py", line 1714, in __init__
    Parameter.__init__(self, param_decls, type=type, **attrs)
TypeError: __init__() got an unexpected keyword argument 'shell_complete'

I tend to see vanguard transaction events where two dividends accumulate and sweep into a money market account.

i.e. let's say you have two dividend transactions:

<INCOME><INVTRAN>
<FITID>123888456<DTTRADE>20210301160000.000[-5:EST]<DTSETTLE>20210301160000.000[-5:EST]
<MEMO>DIVIDEND PAYMENTDIVIDEND PAYMENT</INVTRAN>
<SECID><UNIQUEID>92206C300<UNIQUEIDTYPE>CUSIP</SECID>
<INCOMETYPE>DIV<TOTAL>50.00<SUBACCTSEC>CASH<SUBACCTFUND>CASH</INCOME>

<INCOME><INVTRAN>
<FITID>123777456<DTTRADE>20210301160000.000[-5:EST]<DTSETTLE>20210301160000.000[-5:EST]
<MEMO>DIVIDEND PAYMENTDIVIDEND PAYMENT</INVTRAN>
<SECID><UNIQUEID>92206C821<UNIQUEIDTYPE>CUSIP</SECID>
<INCOMETYPE>DIV<TOTAL>100.00<SUBACCTSEC>CASH<SUBACCTFUND>CASH</INCOME>

That pool together to result in this purchase of a money-market asset:

<BUYTYPE>BUY</BUYMF><BUYMF><INVBUY><INVTRAN>
<FITID>123999456<DTTRADE>20210301160000.000[-5:EST]<DTSETTLE>20210301160000.000[-5:EST]
<MEMO>MONEY FUND PURCHASE</INVTRAN>
<SECID><UNIQUEID>922906300<UNIQUEIDTYPE>CUSIP</SECID>
<UNITS>150.00<UNITPRICE>1.0<TOTAL>-150.00<SUBACCTSEC>CASH<SUBACCTFUND>

Currently it results in:

2021-03-01 * "MONEY FUND PURCHASE" "[VMFXX] Vanguard Federal Money Market Fund"
  file_account: "Assets:Vanguard:Brokerage"
  Assets:Vanguard:Brokerage:VMFXX   150.00 VMFXX {1.0 USD}
  Assets:Vanguard:Brokerage:USD    -150.00 USD            

2021-03-01 * "DIVIDEND PAYMENTDIVIDEND PAYMENT" "[VSBSX] Vanguard Short-Term Treasury Index - Admiral Shares"
  Assets:Vanguard:Brokerage:USD               50.00 USD
  Income:Vanguard:Brokerage:VSBSX:Dividends  -50.00 USD

2021-03-01 * "DIVIDEND PAYMENTDIVIDEND PAYMENT" "[VLGSX] Vanguard Long-Term Treasury Index - Admiral Shares"
  Assets:Vanguard:Brokerage:USD               100.00 USD
  Income:Vanguard:Brokerage:VLGSX:Dividends  -100.00 USD

This is functional, but feels rather odd to have three transactions, and requires that there's an imaginary account of Assets:Vanguard:Brokerage:USD for everything to work. Is it possible for the importer to be a bit smart and glob together these events based on the fact that they all have the same timestamp (20210301160000.000)? the FITID's are also similar-ish (the first three and last three digits usually are all the same between these events as shown above). Ideally thinking the generated result should look like:

2021-03-01 * "MONEY FUND PURCHASE & DIVIDEND PAYMENTDIVIDEND PAYMENT & DIVIDEND PAYMENTDIVIDEND PAYMENT" "[VMFXX] Vanguard Federal Money Market Fund & [VSBSX] Vanguard Short-Term Treasury Index - Admiral Shares & [VLGSX] Vanguard Long-Term Treasury Index - Admiral Shares"
  file_account: "Assets:Vanguard:Brokerage"
  Assets:Vanguard:Brokerage:VMFXX   150.00 VMFXX {1.0 USD}
  Income:Vanguard:Brokerage:VSBSX:Dividends  -50.00 USD
  Income:Vanguard:Brokerage:VLGSX:Dividends  -100.00 USD

Feel free to close this idea if it sounds too risky/complicated, just trying to brainstorm anything that comes to mind.

1. "Cash Withdrawal" entries should have 2 decimal places even for whole numbers but are currently missing. This affects automatic numerical display inference in Fava
2. Entries should mirror numerical precision from exported Excel bank source, which are to 2 decimal places. But this seems to be rounded down (now) plus the numerical precision seems to go out to several lengthy number of places (which also somehow crashes fava)
3. Importer outputs a warning message to terminal line when ran: "WARNING *** file size (92598) not 512 + multiple of sector size (512)"
4. Last minor nit tweak: Possible to output the larger positive numberical figures, e.g., 1,000, as it is, without commas (i.e, 1000)? Commas rendering should be something adjustable in fava, etc., but not part of the raw beancount output?

In the examples/ directory, the my-smart.importer references the smart_importer package that doesn't exist. Perhaps it was removed in the past?

This .import file is referenced in the import.sh script which is mentioned in the README to run the examples.

Additionally, running bean-identify my.import transactions.qfx does not seem to match any Importers.

Qualified Dividend is another Schwab Brokerage CSV action that should be supported by this library.

Example Record

"Date","Action","Symbol","Description","Quantity","Price","Fees & Comm","Amount"
"02/01/2023","Qualified Dividend","GIS","GENERAL MILLS INC","","","","$0.54"

Should just be treated as a reds_importer dividend type.

Let me know! I could add this myself but I won't be able to get to it for a bit! Also blocked by #59.

I use Fidelity Checking/Savings accounts. These are investment accounts that act like banking accounts. I'd like to use the Smart Importer to manage these. I have hacked together a quick solution which is incomplete but currently working for me. There are two main changes that need to be made to libtransactionbuilder/investments.py:

1. Add existing_entries to extract method signature

@@ -364,7 +366,7 @@ class Importer(importer.ImporterProtocol):
         """For custom importers to override"""
         return []
 
-    def extract(self, file):
+    def extract(self, file, existing_entries=None):
         self.initialize(file)
         counter = itertools.count()
         new_entries = []

2. Drop the second posting for non investment type transactions (would need to add a few more types here).

@@ -264,6 +264,8 @@ class Importer(importer.ImporterProtocol):
         if ot.type in ['income', 'dividends', 'capgainsd_st', 'capgainsd_lt']:  # cash
             data.create_simple_posting(entry, config['cash_account'], ot.total, self.currency)
             data.create_simple_posting(entry, target_acct, -1 * ot.total, self.currency)
+        elif ot.type in ['dep', 'cash']:  # banking
+            data.create_simple_posting(entry, main_acct, units, ticker)
         else:
             data.create_simple_posting(entry, main_acct, units, ticker)
             data.create_simple_posting(entry, target_acct, -1 * units, ticker)

I'm not entirely sure how best to go about this since this has the potential to break existing expectations. Two options come to mind.

1. Drop the posting if "target_acct" has not been configured (making optional keys might get messy)
2. Provide some kind of toggle to exclude posting account (but this would seem to break from convention).

Thoughts?

Hi Again,

One of my accounts (Wise), doesn't export OFX files, but there is a tool to convert the CSV files to OFX files ({ofxstatement}). Unfortunately, the Reds OFX importers don't seem to be able to read the converted files.

I currently use {bean count-import}, which is able to import these OFX files after removing an empty <DTASOF /> tag from the end of the file - see here and here for more details. However, I prefer the logic of your framework and like the idea of using your plugins, so I'm trying to make the switch, but this is a sticking point.

I get the error below.

❯ bean-extract my.import 2021_Wise-CAD.ofx
ERROR:root:Importer personal_importers.wise_cad.Importer.identify() raised an unexpected error: list index out of range
Traceback (most recent call last):
  File "/Users/usr/.pyenv/versions/3.10.4/envs/personal-finance/lib/python3.10/site-packages/beancount/ingest/identify.py", line 63, in find_imports
    matched = importer.identify(file)
  File "/Users/usr/.pyenv/versions/3.10.4/envs/personal-finance/lib/python3.10/site-packages/beancount_reds_importers/libreader/reader.py", line 25, in identify
    self.initialize_reader(file)
  File "/Users/usr/.pyenv/versions/3.10.4/envs/personal-finance/lib/python3.10/site-packages/beancount_reds_importers/libreader/ofxreader.py", line 20, in initialize_reader
    self.ofx = ofxparse.OfxParser.parse(open(file.name))
  File "/Users/usr/.pyenv/versions/3.10.4/envs/personal-finance/lib/python3.10/site-packages/ofxparse/ofxparse.py", line 396, in parse
    ofx_file = OfxPreprocessedFile(file_handle)
  File "/Users/usr/.pyenv/versions/3.10.4/envs/personal-finance/lib/python3.10/site-packages/ofxparse/ofxparse.py", line 183, in __init__
    tag_name = re.findall(r'(?i)<([a-z0-9_\.]+)>', token)[0]
IndexError: list index out of range
;; -*- mode: beancount -*-

where my importer has the following code

"""Wise ofx importer for beancount."""

from beancount_reds_importers.libreader import ofxreader
from beancount_reds_importers.libtransactionbuilder import banking


class Importer(banking.Importer, ofxreader.Importer):
    def custom_init(self):
        if not self.custom_init_run:
            self.max_rounding_error = 0.04
            self.account_number_field = 'account_id'
            self.filename_pattern_def = '.*[Ww]ise'
            self.custom_init_run = True

Based on the trace it looks like it might be an issue with the {ofxparse} package and possible empty tags, as well as this issue`, so I appreciate this may not be something that can be fixed here, but I have only moderate python experience so my ability to diagnose a problem and hack a solution is limited, and I figured I may be missing something.

If it would be useful, I can try and put together a sample OFX file for a reprex.

Thanks,

First, thank you so much for your blog, and your importers and everything. I've been using beancount for 3 years now, and its just gotten too much for me to input stuff. Your blog inspired me to try again and to be a lot smarter about it this time.

I had an idea for a pdfreader module. My thought was to utilize the pdfplumber package, to extract tables from pdfs, and something like this filter to capture all the data outside of the tables as well. I'd like to also search for everything that looks like a date in outside-of-table information, and return those in a list. And then everything that looks like a currency outside of the tables as well in a list. Perhaps we can put all that outside-of-table information into a metadata table and pass all of the data on as a multi table format to downstream formatters.

My primary concern right now is reading my paystub pdf, but I think it could also be useful for parsing pdf statements. Thoughts?

Filing_account has no effect for either banking or investment importers.
Using smart_importer together with the patched bean-extract leads to file names related to the first posting in each transaction entry, which is not necessarily the asset account.