Comments (4)
Do I understand correctly that you would like to just use the beancount-import ofx source to non-interactively generate journal entries from ofx files?
And have it skip previously imported entries but not attempt to do any matching against manually-entered transactions or transactions from other sources, and not attempt to predict unknown accounts?
from beancount-import.
I would like to follow the same vanilla beancount
patterns: identify
extract
archive
and generate
test
. The benefit of the pattern is that there is determinism in the process.
With beancount
importers, one can keep the same exact workflow, but just replace the extract
step with beancount-import
. With beancount-import
sources, beancount-import
handles the identify
part when one configures the data_sources
, and the extract
step is handled semi-automatically. I am not as clear about how to achieve the archive
, generate
, and test
steps.
For archive
, beangulp
archives to the associated account. I don't think this mechanic exists in beancount-import
, nor are the importers necessarily associated with a single account.
As you mentioned, generate
and test
could possibly be accomplished by generating and comparing against a non-interactive, deterministic output, even if the file is incomplete (e.g. the output of beangulp.examples.ofx
. One input file + one importer = one output. And the input and output files can serve as the test cases as the importers change.
from beancount-import.
I'm still not sure I understand what your objective is.
The identify
step is not really in scope for beancount-import, and nothing in beancount-import currently handles that. My understanding is that the idea behind the identify step is that you will manually browse to your bank's website, and click on links to download files. All of these files will get dumped into the same download directory, and then you use the identify tool on that directory to avoid having to manually worry about the file names. For my own use I have instead relied on automating the download process (see https://github.com/jbms/finance-dl) so that files are all automatically downloaded with a suitable organization, and the identify step is not needed. Still, I can see that if you are downloading the files manually, the identify step may be useful. As beancount-import does not have any code to do that, though, you will have to either write it yourself or rely on something existing in beangulp.
beancount-import basically addresses just the extract
step. I'm still not clear, though, on whether you want to use beancount-import normally through the web UI, or you want some entirely non-interactive import as with beangulp.
The archive
step is also not applicable to beancount-import: beancount-import relies on metadata on the imported entries to determine what has already been imported and avoid creating duplicate entries. Therefore, there is no need to store the source files in a separate location after they are imported, and in fact beancount-import relies on them being present as normal inputs in order to detect invalid references to source data in transaction metadata.
As for generating test cases from your private data, there is no specific command-line tool for doing that, but the existing test framework essentially supports what you want. Take a look at ofx_test.py. You could very easily create test cases from additional data files. You can also potentially use ofx_sanitize.py to sanitize your source data so that you can create test cases that can be shared without leaking your private information.
from beancount-import.
Ultimately, I would like to use beancount-import
normally through the web UI. The question is more to do with how to marry the two similar, but also sufficiently different, workflows effectively.
Concretely, I'm thinking the workflow could look like this:
- Download files (with arbitrary names), either manually or programmatically, to a central directory
identify
to determine which files correspond with which importers- This step isn't as critical, but is helpful information and serves as a manual check
- Could also just be something displayed in the UI maybe
extract
viabeancount-import
UIarchive
to move file from central directory to an archive directory with proper naming (beangulp extract
uses account name and date, which generally makes sense, although I'm not exactly sure if there is an "account" analogue forbeancount-import
)- As mentioned, this does not exist in
beancount-import
, and should be relatively straightforward to replicate, but would essentially be like maintaining another (albeit lightweight)beangulp
/beancount.ingest
importer just for archiving
- As mentioned, this does not exist in
generate
to generate a deterministic.beancount
output (inbeangulp
case, this is basically just theextract
output)- Admittedly I am not too familiar with
beancount-import
tests, but I thinkbeancount-import
is a supersetbeangulp
/beancount.ingest
's? Is there a way to generate the test cases programmatically? - End result is something like this?
. └── archive └── Assets └── <path> └── <to> └── <account> ├── 2021-05-18.statement.ofx ├── 2021-05-18.statement.ofx.beancount └── <other beancount-import outputs>
- Admittedly I am not too familiar with
test
to automatically compare all files in a given directory against theirgenerate
output
I think for generic beangulp
/beancount.ingest
importers, I can follow the exact workflow using the vanilla CLI, just replacing the extract
step with beancount-import
using the web UI. That leaves the ofx beancount-import
source. Although seems like there is probably a better solution, e.g. maybe implementing some archiving capability into beancount-import
and building on the testing capabilities to automatically generate a deterministic test cases from a file and automatically testing against that?
from beancount-import.
Related Issues (20)
- BUYOTHER and SELLOTHER OFX transaction types are not recognized HOT 1
- Amazon import fails when Price is empty
- Add multiple postings to imported transaction HOT 5
- Verifying that I have receipts for all transactions HOT 1
- Mark an entry as incomplete HOT 1
- transactions matched that shouldn't be HOT 2
- Sklearn get_feature_names() deprecated HOT 2
- GenericImporterSource - which data/field is used for the source_desc field? HOT 3
- Fidelity OFX has two CUSIP per ticker HOT 4
- Is this project dead? HOT 1
- local install tornado issue HOT 2
- Tests are broken after release of Beancount 2.3.6 HOT 1
- Transactions with foreign transactions fees shows as 2 separate transactions with same QFX FITID, only one being imported. HOT 1
- Importer doesn't seem to recognize some transactions from QFX from Merrill Edge HOT 1
- schwab_csv importer raises AssertionError because CSV has no title line HOT 1
- Request for PyPI release HOT 4
- Mark transaction as non-interactive
- Honor __duplicate__ from bean-extract
- File permissions are lost whenever a journal file is written to HOT 4
- Does not properly clear transactions with multiple postings on the main account HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from beancount-import.