apauley / hledger-flow Goto Github PK

Right now the input statements are converted into journal files sequentially.

Change the import logic to process them in parallel.

The release tarballs are generated in the TravisCI builds by running the script bin/release-tarball.

Add a README.org (or README.md) to this release tarball, so that users who download it have some instructions.

The README should briefly mention how to verify the binary (shasum -c sha256-hledger-flow_*) and where to copy it so that it is in your PATH

Is your feature request related to a problem? Please describe.

I want to import historic currency exchange rates and stock prices into hledger-flow.
Unfortunately using a construct script doesn't work because hledger-flow filters out and only retains transactions, not prices, see

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.12.2.99 linux x86_64 ghc 8.6

Is this the latest version?

No

Our Example Statements Repository

FYI, we have a repo with some example transactions which you can use to run hledger-flow on:
https://github.com/apauley/hledger-flow-example

Can you give examples of what you would like by running hledger-flow on these files?

No, they do not contain prices.

Describe the solution you'd like
It should be possible to import prices, not just transactions.
First step would be to have an equivalent of the construct script that can output only prices in a format that hledger understands.
A second step would be to have support for csv import rules that can deal with the various date formats and convert to hledger format.

Describe alternatives you've considered

I wrote a script that generates .journal files and puts them into a _manual_ folder.
This is a bit of a hack, because it is not part of the usual hledger-flow import, I have to run it separetely.
What I ended up having is this:

cat import/edwin/_manual_/run.sh
#/bin/sh
set -eu
cd prices;
(for i in $(ls -1d */ | cut -f1 -d/); do
    for j in ${i}/1in/*/*; do
        ${i}/construct "${j}" - X "${i}" X
    done
done) >pricelist
cd ..
for i in 2017 2018 2019; do
    echo 'D £1,000.00' >${i}/pre-import.journal
    grep $i prices/pricelist | sed -e 's/"EURGBP"/€/' >>${i}/pre-import.journal
done

cat import/edwin/_manual_/prices/construct
#!/bin/sh
# this is very slow, should reimplement in python or something else
set -eu
IFS=^
cut -f1-2 --output-delimiter=^ <"$1" | while read D PRICE; do
    D=$(date +%F -d "$D")
    echo P $D \"$4\" $PRICE
done

tail import/edwin/_manual_/prices/EURGBP/1in/2019/EURGBP
Jan 11, 2019 	0.8925 	0.9020 	0.9064 	0.8921 	-1.05%
Jan 10, 2019 	0.9020 	0.9026 	0.9063 	0.9011 	-0.06%
Jan 09, 2019 	0.9025 	0.8997 	0.9044 	0.8973 	0.33%
Jan 08, 2019 	0.8995 	0.8980 	0.9010 	0.8949 	0.18%
Jan 07, 2019 	0.8979 	0.8951 	0.8995 	0.8948 	0.20%
Jan 06, 2019 	0.8961 	0.8952 	0.8967 	0.8949 	0.09%
Jan 04, 2019 	0.8953 	0.9019 	0.9032 	0.8942 	-0.75%
Jan 03, 2019 	0.9021 	0.9000 	0.9133 	0.8986 	0.28%
Jan 02, 2019 	0.8996 	0.8991 	0.9034 	0.8981 	0.07%
Jan 01, 2019 	0.8990 	0.8985 	0.9057 	0.8953 	0.00%

Additional context

Obtaining the prices themselves is another matter, but that is independent of hledger-flow.
What I did was just copy/paste the tables from here, but this could easily be automated by something that fetches the webpage and parses it.

For example investment stock prices can be fetched from (using a random example), you can view 1 year ranges:
https://markets.ft.com/data/funds/tearsheet/historical?s=GB0033053710:GBX

Currencies can be found in a similar place, see:
https://github.com/barrucadu/hledger-scripts/blob/master/market-prices/market-prices.py

Stock prices can be found here (again using a random example)
https://uk.finance.yahoo.com/quote/GOOG/history/?guccounter=1

I haven't done anything interesting with the prices yet, other than being able to see how much I own were I to convert everything into GBP right now, so having daily price granularity is probably an overkill, but it was the easiest way to obtain the data.

(in v0.11.1.2)

I'm experiencing an error importing a CSV, and am at a loss as to whether the problem is in hledger or hledger-flow or my CSV file, or my rules file. The pstateInput = "..." part in the error below contains my whole CSV input file, which I cannot quote here because it is huge and contains PII, so i just put ...

2019-05-08 15:10:48.668158024 EEST	hledger-flow End:   importing '2-preprocessed/2018/2018.csv' using rules file 'bank-account.rules' ExitFailure 1 (0.081442287s)
hledger: user error (CSVError (ParseErrorBundle {bundleErrors = TrivialError 14 (Just (Tokens (59 :| []))) (fromList [Tokens (44 :| []),Label ('e' :| "nd of line"),EndOfInput]) :| [], bundlePosState = PosState {pstateInput = "...", pstateOffset = 0, pstateSourcePos = SourcePos {sourceName = "import/personal/bank/account/2-preprocessed/2018/2018.csv", sourceLine = Pos 1, sourceColumn = Pos 1}, pstateTabWidth = Pos 8, pstateLinePrefix = ""}}))

hledger-flow: an external process exited with exit code 1.

The CSV file in question loads without error in Libreoffice

In order to rule out whether it's a quoting issue, or a separating character issue, I'd need to know what quoting character and / or separating character was in use. The error and log statement do not say.

The (probably incomplete) rules file:

skip 7

fields Buchungstag,Wertstellung,Buchungstext,Auftraggeber_Beguenstigter,Verwendungszweck,Kontonummer,BLZ,Betrag_EUR,Glaeubiger_ID,Mandatsreferenz,Kundenreferenz

account1 Personal:Assets:DE:Bank:Account

amount EUR %Betrag_EUR

date %Wertstellung
date-format %d.%m.%Y

currency EUR

There are also non ascii characters, but that should be no problem in a UTF8 file

<LATIN SMALL LETTER A WITH DIAERESIS> -> ä
<LATIN SMALL LETTER O WITH DIAERESIS> -> ö
<LATIN SMALL LETTER U WITH DIAERESIS> -> ü

Run hledger --version on start of execution to determine if hledger is in the path.

Exit with a helpful error message if hledger cannot be found.

The error message we currently see when running hledger-flow on a system that doesn't have hledger installed isn't very helpful:

hledger: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

Keep the hledger version number around so that it can be used by other parts of the program.
It will be helpful for example to show this version number when printing out the program options (issue #11).

We probably require a specific version (or later) of hledger, although I don't know what the earliest version of hledger is that will work with hledger-flow.

Description

In the situation where one has yyyy-opening.journal and yyyy-closing.journal for a specific year,
running one of the hledger built-in financial reports on one specific year, the user will be faced with empty reports

$ hledger bal 2017-include.journal 
--------------------
                   0

because hledger-flow adds an include to yyyy-closing.journal in the yyyy-include.journal, which effectively brings all the counters back to zero.

Expectation

I would expect hledger <any report> -f yyyy-include.journal to just work

Solution

The yyyy-opening.journal include stays in yyyy-include.journal, but yyyy-closing.journal include needs to move to the all-years.journal file, as evidenced by the all.journal in the full-fledged-hledger project. This ensures that reports work on a per-year yyyy-include.journal basis, and that reports also work on an aggregated all-years.journal basis, with the important caveat that the very last yyyy-closing.journal must not be part of the all-years.journal (even if one such file is present on the file system), otherwise we encounter the same problem of empty reports on an aggegated all-years.journal basis.

I hope that makes sense, it's a bit tricky to explain.

Version and Runtime Information

This include behaviour has been present in hledger-flow as far back as I can remember using it.

Additional context

I only found this bug because I'm trying to use hledger-flow import for my imports, while using full-fledged-hledger's export.hs for my reports, and I got confused that I was getting empty reports.

Is your feature request related to a problem? Please describe.

As my full import takes longer and longer, it starts making sense to restrict the years to import. Currently it imports as far back as it finds years. My top 5 time consumers during full import are from past year.

I would need to import only the current year, and I could revert to full import when I make
a change that influences past journal generation, but only pay that extra time when I do the full import.

I'm reluctant to put actual stats here, because my files contain personal info, but If you need it I can give you an idea of the top 5, which each take a couple minutes, and are all before 2020.

Add a command-line switch that will cause the values of all options to be printed.

This will include the basedir being used, and the value of any of the other command-line flags that can be passed in, e.g. the current verbosity.

If the option is specified the program will import statements sequentially. The same for report generation.

Parallel processing should still be the default behaviour.

Describe the bug
I cheat a bit and add, next to each .journal file in 3-journal, a .png of a related invoice, and a .json of the google text detection api response. These extra files used to be ignored by hledger-flow import, now they get added to the yearly include, causing hledger to fail:

  |
6 | !include 3-journal/2019/2019-05-31-000.png
  |                                           ^
SourcePos {sourceName = "SCRUBBED/Finance/import/personal/wallet/cash/2019-include.journal", sourceLine = Pos 6, sourceColumn = Pos 43} reading SCRUBBED/Finance/import/personal/wallet/cash/3-journal/2019/2019-05-31-000.png:
SCRUBBED/Finance/import/personal/wallet/cash/3-journal/2019/2019-05-31-000.png: hGetContents: invalid argument (invalid byte sequence)

Version and Runtime Information

hledger-flow-v0.14.2.0

Is this the latest version? yes
Please confirm your issue using the latest version of hledger-flow, maybe it has already been fixed.

Also include the runtime options of the command you ran, e.g:

The enable-future-rundir option is now the default, no need to specify it. This option is currently being ignored and will be removed in future.
RuntimeOptions {baseDir = "SCRUBBED/Finance/", importRunDir = "import/personal/wallet/cash/", importStartYear = Nothing, onlyNewFiles = False, hfVersion = "hledger-flow 0.14.2.0 linux x86_64 ghc 8.10", hledgerInfo = HledgerInfo {hlPath = FilePath "SCRUBBED/Finance/hledger", hlVersion = "hledger 1.20.2"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,10], versionTags = []}}, verbose = False, showOptions = True, sequential = False}
Collecting input files from SCRUBBED/Finance/import/personal/wallet/cash/
Found 100 input files in 0.024561536s. Proceeding with import...

To Reproduce

Add files in 3-journal as part of a construct script

Expected behavior

One of:

• ignore non- .journal extensions in 3-journal
• make the list of extensions that will be included in the yearly include configurable...

First, thank you for hledger-flow! I really appreciate the opinionated guidance on structure and the help on the non-destructive workflow.

I have a really basic question, but which I really couldn't find any answer about, nor here, nor on hledger-flow-example, nor on full-fledged-hledger:

Where do you put "meta"/generic statements that apply to all your hledgering, such as:

account assets:bank:savings
account liabilities:bank:creditcard

alias USD = $
commodity 
  format  1,000.00 $
D 1,000.00 $

include import/prices/YEAR.prices

In my initial, single-file journal workflow, I had a meta.journal which included all this and was included from the main journal, but here, I'm struggling to find where it would be appropriate to include it.

Do you use a main journal file besides those created by hledger-flow?
Where does LEDGER_FILE env constant points to? (= where does hledger defaults to without -f?)
That seems important to be able to use hledger without -f or outside its main directory, and so that you can use hledger add to add manual transactions directly to the right place (in owner/_manual_/2021/post-import.journal, I guess?)

Note: I think this basic question (and other similar ones) could (have) be(en) solved by adding more files to hledger-flow-example: a _manual_ directory with some basic examples in pre-import.journal and post-import.journal (I'm not sure what goes in each), an import/prices/2016.prices example and where it is included, some file with the meta statements I'm mentioning, etc. (If it can help you, I can do a PR with these files as soon as I understand the recommended practice on the matter.)

I'd ideally also include on hledger-flow-example a .gitignore file: I'm guessing it is not really necessary to version control files that are entirely generated and transitional (e.g., 2-preprocessed/...)? Do you actually version control both source (1-in/*.csv) and generated (3-journal/*.journal) files?

And a README listing some typical commands you would run within this workflow could be useful as well :-)

Related to #14

The --verbose flag is currently enabled on each subcommand, e.g.:

$ hledger-flow import --verbose
2019-04-27 11:03:05.808182368 SAST	hledger-flow Starting import

or

$ hledger-flow report --verbose
2019-04-27 11:03:33.90158193 SAST	hledger-flow Something will be here Real Soon Now (tm)

Change this so that the verbose flag is part of the main options, similar to --help and --version.

However, unlike --help and --version, the verbose flag should not just print some output and exit.
Instead, it should continue to expect one of the subcommands to be supplied, and then run the relevant subcommand with verbose output enabled.

Evaluate all command-line options.
Design a good user interface for both interactive use and scripting.

Does the names of the options make sense? Can we anticipate possible name clashes for future options?

Should some subcommands options be moved to the main options, if they apply to all subcommands?

in bank1.journal, imported from bank1.csv

2018/11/09 (6138507615589995706)
    Personal:Assets:BANK1:Checking:EUR      EUR -11,22 = EUR 33,44
    Personal:Assets:BANK2:Girokonto:EUR

in bank2.journal, imported from bank2.csv

2018/11/09 (6138507615589995706)
    Personal:Assets:BANK2:Girokonto:EUR        EUR11,22
    Personal:Assets:BANK1:Checking:EUR       EUR-11,22

When I then run hledger-ui, I get an balance assertion error, because it sees two debits of EUR-11,22, not a single one, within the same day. I added the transaction code to try to make hledger-ui see one unique transaction, to no avail. It seems to work that way by design, which makes it difficult to handle transfers between my accounts in the hledger-flow world.

I tried to use virtual postings for bank2, to no avail.

What is the correct way to do this in the hledger world, and most importantly in the hledger-flow world?

Approach 1: Remove problematic line in the bank2 preprocess script

Probably the simplest approach, the bank2 preprocess script could have a grep -v <txid we want to ignore>, so that the transaction ends up in only one journal.

Problem: In the bank subdirectory, there is a CSV line for the transaction in question in 1-in, but no corresponding transaction in 3-journal. It feels like we're rewriting history.

Additionally, we now lose some information, such as when the day on which the money leaves bank1 is different from the day on which the money arrives in bank2.

Approach 2: Comment out the problematic journal entry using a bank2 construct script

Problem: You need to switch to using a construct file, you lose the ability to use a .rules file.

My current approach is Approach 1, since in my case bank2 uses a .rules file.

Have you encountered this situation, and how did you resolve it?

Verbose logs for the external script hooks and other shell commands currently do not log the exit status of the command:

2019-04-05 23:27:36.499626412 SAST	hledger-makeitso Begin: executing 'import/gawie/bogart/cheque/preprocess' on 'import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv'
2019-04-05 23:27:36.531189098 SAST	hledger-makeitso End:   executing 'import/gawie/bogart/cheque/preprocess' on 'import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv' (0.031460856s)

Log the exit code as well, so that the logs look something like:

2019-04-05 23:27:36.499626412 SAST	hledger-makeitso Begin: executing 'import/gawie/bogart/cheque/preprocess' on 'import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv'
2019-04-05 23:27:36.531189098 SAST	hledger-makeitso End:   executing 'import/gawie/bogart/cheque/preprocess' on 'import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv' ExitSuccess (0.031460856s)

This will probably require us to change usages of procs into proc.

Documentation that may prove useful:
https://hackage.haskell.org/package/turtle-1.5.13/docs/Turtle-Prelude.html#v:proc

The config file location should be the basedir, but the location could also be specified via a command-line option.

The config file would support the same options as the command-line options.

In all cases the options specified via the command-line would override any option present in a config file.

Describe the bug
Using: cabal install hledger-flow
Receive:
Resolving dependencies...
Build profile: -w ghc-8.10.5 -O1
In order, the following will be built (use -v for more details):

• gitrev-1.3.1 (lib) (requires download & build)
• hostname-1.0 (lib:hostname) (requires download & build)
• managed-1.0.8 (lib) (requires download & build)
• mwc-random-0.15.0.1 (lib) (requires download & build)
• optional-args-1.0.2 (lib) (requires download & build)
• path-0.8.0 (lib) (requires download & build)
• profunctors-5.6.2 (lib) (requires download & build)
• semigroupoids-5.3.5 (lib) (requires download & build)
• system-filepath-0.4.14 (lib:system-filepath) (requires download & build)
• path-io-1.6.2 (lib) (requires download & build)
• foldl-1.4.11 (lib) (requires download & build)
• system-fileio-0.3.16.4 (lib) (requires download & build)
• turtle-1.5.22 (lib) (requires download & build)
• hledger-flow-0.14.3.0 (lib) (requires download & build)
• hledger-flow-0.14.3.0 (exe:hledger-flow) (requires download & build)
  Downloading hostname-1.0
  --snipped--
  Completed turtle-1.5.22 (lib)
  Starting hledger-flow-0.14.3.0 (lib)
  Building hledger-flow-0.14.3.0 (lib)

Failed to build hledger-flow-0.14.3.0.
Build log (
/home/reed/.cabal/logs/ghc-8.10.5/hledger-flow-0.14.3.0-fcb2ef806d78720166fe03f1b62892828c13b32c2d2ad0712f200a12cd6af4f9.log
):
Configuring library for hledger-flow-0.14.3.0..
Preprocessing library for hledger-flow-0.14.3.0..
Building library for hledger-flow-0.14.3.0..
[ 1 of 15] Compiling Hledger.Flow.DateTime ( src/Hledger/Flow/DateTime.hs, dist/build/Hledger/Flow/DateTime.o, dist/build/Hledger/Flow/DateTime.dyn_o )
[ 2 of 15] Compiling Hledger.Flow.DocHelpers ( src/Hledger/Flow/DocHelpers.hs, dist/build/Hledger/Flow/DocHelpers.o, dist/build/Hledger/Flow/DocHelpers.dyn_o )
[ 3 of 15] Compiling Hledger.Flow.PathHelpers ( src/Hledger/Flow/PathHelpers.hs, dist/build/Hledger/Flow/PathHelpers.o, dist/build/Hledger/Flow/PathHelpers.dyn_o )
[ 4 of 15] Compiling Hledger.Flow.Import.Types ( src/Hledger/Flow/Import/Types.hs, dist/build/Hledger/Flow/Import/Types.o, dist/build/Hledger/Flow/Import/Types.dyn_o )
[ 5 of 15] Compiling Hledger.Flow.Types ( src/Hledger/Flow/Types.hs, dist/build/Hledger/Flow/Types.o, dist/build/Hledger/Flow/Types.dyn_o )
[ 6 of 15] Compiling Hledger.Flow.Logging ( src/Hledger/Flow/Logging.hs, dist/build/Hledger/Flow/Logging.o, dist/build/Hledger/Flow/Logging.dyn_o )
[ 7 of 15] Compiling Hledger.Flow.BaseDir ( src/Hledger/Flow/BaseDir.hs, dist/build/Hledger/Flow/BaseDir.o, dist/build/Hledger/Flow/BaseDir.dyn_o )
[ 8 of 15] Compiling Hledger.Flow.Common ( src/Hledger/Flow/Common.hs, dist/build/Hledger/Flow/Common.o, dist/build/Hledger/Flow/Common.dyn_o )

src/Hledger/Flow/Common.hs:46:46: error:
• Not in scope: type constructor ‘Rel’
• In the quasi-quotation: [relfile|hledger|]
|
46 | maybeH <- Path.findExecutable [relfile|hledger|]
| ^^^^^^^^^

cabal: Failed to build hledger-flow-0.14.3.0 (which is required by
exe:hledger-flow from hledger-flow-0.14.3.0). See the build log above for
details.
Formating of above is with carats as positioned here, starting at 'h':
Executable [relfile|hledger|]
.............................^^^^^^^^^
Things that work
Can confirm 'hledger' is on path: from same command prompt, 'which hledger' (/home/username/.cabal/bin/hledger), 'hledger --help', 'hledger add' and 'which cabal' (/home/username/.local/bin/cabal) all respond correctly.

Have successfully installed other hledger extensions from same command prompt. For example: 'cabal install hledger-diff' and 'cabal install hledger-iadd' are successfully added.
Expected result
Was expecting the usual: 'Symlinking 'hledger-flow' to …
Install OS and Location
Using Ubuntu v18.04 up to date daily. Command prompt at Gnome terminal. All install is local at '/home/username/' at user $ prompt.

Thank you.

Describe the bug
A clear and concise description of what the bug is.

Lowest level auto generated files use relative import starting from the top import path. All other autogenerated journal files use the correct relative path. This does not happen is hledger-flow is started in the same folder that import is defined.

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.12.3.0 linux x86_64 ghc 8.6

Also include the runtime options of the command you ran, e.g:

[max@thinki flow]$ ./hledger-flow --show-options import ledgers/
RuntimeOptions {baseDir = FilePath "ledgers/", hfVersion = "hledger-flow 0.12.3.0 linux x86_64 ghc 8.6", hledgerInfo = HledgerInfo {hlPath = FilePath "/usr/bin/hledger", hlVersion = "hledger 1.14.2"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,6], versionTags = []}}, verbose = False, showOptions = True, sequential = False}
Collecting input files...
Found 2 input files in 0.004298132s. Proceeding with import...
Imported 2 journals in 0.270081829s

To Reproduce

1. Download the example repository
2. ./hledger-flow import hledger-flow-example
3. ./hledger-flow report hledger-flow-example

=== Begin Error: hledger --file all-years.journal --period 2016 balance --pretty-tables --quarterly --flat --no-total transfer ===
External command:
/usr/bin/hledger --file hledger-flow-example/all-years.journal --period 2016 balance --pretty-tables --quarterly --flat --no-total transfer
Exit code 1
Error output:
hledger: in file included from /home/max/test/hledger-flow-example/all-years.journal,
in file included from /home/max/test/hledger-flow-example/import/2016-include.journal,
in file included from /home/max/test/hledger-flow-example/import/gawie/2016-include.journal,
in file included from /home/max/test/hledger-flow-example/import/gawie/bogart/2016-include.journal,
/home/max/test/hledger-flow-example/import/gawie/bogart/cheque/2016-include.journal:4:101:
  |
4 | !include hledger-flow-example/import/gawie/bogart/cheque/3-journal/2016/123456789_2016-03-30.journal
  |                                                                                                     ^
No existing files match pattern: hledger-flow-example/import/gawie/bogart/cheque/3-journal/2016/123456789_2016-03-30.journal


=== End Error: hledger --file all-years.journal --period 2016 balance --pretty-tables --quarterly --flat --no-total transfer ===

Expected behavior

I can run hledger-flow in a different folder as specified in the documentation.

Desktop (please complete the following information):

$ uname -a
Linux thinki 5.1.4-arch1-1-ARCH #1 SMP PREEMPT Wed May 22 08:06:56 UTC 2019 x86_64 GNU/Linux

First off, thanks for a wonderful tool!

Describe the bug
If I delete a file in a 1-in directory, re-running hledger-flow import does not remove the corresponding files in the 2-preprocessed and 3-journal directories.

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.14.4 linux x86_64 ghc 9.0 UNKNOWN

I know this is not the latest version of hledger-flow, sorry. I looked at the diff between 0.14.4 and 0.15, and I don't think anything related to this has been changed. Let me know if it's really important to test with the latest version of hledger-flow, and I'll figure out how to update.

Also include the runtime options of the command you ran, e.g:

$ hledger-flow --show-options import

To Reproduce

I can reproduce this with a fresh clone of https://github.com/apauley/hledger-flow-example:

$ rm import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv

$ hledger-flow import . --show-options
RuntimeOptions {baseDir = "/home/jeremy/src/github.com/apauley/hledger-flow-example/", importRunDir = "./", importStartYear = Nothing, onlyNewFiles = False, hfVersion = "hledger-flow 0.14.4 linux x86_64 ghc 9.0 UNKNOWN", hledgerInfo = HledgerInfo {hlPath = "/nix/store/lxv7a7ps6k7girbwm50kr3w42ih90v8w-hledger-1.25/bin/hledger", hlVersion = "hledger 1.25, linux-x86_64"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [9,0], versionTags = []}, cores = 4, availableCores = 4}, verbose = False, showOptions = True, sequential = False, batchSize = 200}
Collecting input files from /home/jeremy/src/github.com/apauley/hledger-flow-example/import/
Found 2 input files in 0.001297681s. Proceeding with import...
Wrote include files for 3 journals in 0.002288837s
Imported 2/2 journals in 0.061123857s

$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	deleted:    import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv

no changes added to commit (use "git add" and/or "git commit -a")

Expected behavior

In the example above, I'd expect import/gawie/bogart/cheque/2-preprocessed/2016/123456789_2016-03-30.csv and import/gawie/bogart/cheque/3-journal/2016/123456789_2016-03-30.journal to get deleted as well.

For now, I am working around this by first nuking my 2-preprocessed and 3-journal directories before doing an import:

$ rm -r import/*/*/*/2-preprocessed import/*/*/*/3-journal
$ hledger-flow import .

Desktop (please complete the following information):
I'm running Arch Linux. I installed hledger and hledger-flow with the Nix package manager.

Is your feature request related to a problem? Please describe.

My bank only provides CSV/QIF/QFX statements for recent transactions on current accounts, but not savings accounts.
Historic transactions are available as:

• PDFs containing transaction history with running balance
• PDF for individual transaction events between internal/external accounts, called "advice confirms"
• .xls for stock&shares accounts

Luckily the PDFs are very regular, and by using pdfotext and awk I can "infer" a CSV out of it ... most of the time. .xls can be converted to .csv with ssconvert.

I opened this feature request based on https://news.ycombinator.com/item?id=20024633

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.12.2.99 linux x86_64 ghc 8.6

Is this the latest version?

No.

Our Example Statements Repository

FYI, we have a repo with some example transactions which you can use to run hledger-flow on:
https://github.com/apauley/hledger-flow-example

Can you give examples of what you would like by running hledger-flow on these files?
N/A

Describe the solution you'd like

Would be useful to have:

• some way of determining the sign of a transaction based on a field, I currently have to match on DEBIT/CREDIT in my rules file and swap account1 with account2 accordingly. I tried using amount-in and amount-out, but apparently these cannot appear only inside an if statement. Would be nice if there was a conditional in the CSV rules file that could reverse the sign of a transaction.

• support for account3 in csv rules. investment tracking would require 3 accounts, see https://hledger.org/Track-investments.html with just 2 accounts "we have violated the double-entry principle and created money out of nowhere.", however CSV rules only support 2 accounts currently

• some way of recording unrealized gains/loss when handling multiple currencies, see:
  https://github.com/adept/full-fledged-hledger/wiki/Investments-easy-approach#recording-valuations. This would need something that can generate transactions based on prices. See #69 for support of prices

• support fields with default values in CSV rules. See my awk scripts below, often some fields are empty, meaning that they have the same value as the previous non-empty field (this often happens for dates)

• make it easy to record a currency and an exchange rate for multi-currency transactions inside CSV rule conditionals. Currently I have to calculate the final amount in awk, I think I was not able to do this with just CSV rules due to the lack of support for rounding.

• have some built-in support for tracking transactions between 2 of my own accounts, where I have:
  ** an entry in account X saying I did a transaction with reference Z
  ** an entry in account Y saying I did a transaction with reference Z (if an internal account, missing if external)
  ** an entry in my "advice confirms" transaction list saying I transfered something between 2 accounts with reference Z (sometimes ambigous as it doesn't list both account numbers)
  What I currently do here is to have:

current account -> assets:transfers:%ref
assets:transfers:%ref -> assets:transfers:loyisa:%ref
assets:transfers:loyisa:%ref -> assets:isa:loyisa

This is great for debugging why a certain transaction reference is not balanced (e.g. due to a mistake in my import script, or a missing statement), should there be a builtin way of handling these?

I'd welcome any other suggestion you may have on how to simplify the conversion process I've shown below. What I have now works though, took a while to get here though.

Describe alternatives you've considered
Here is a quick and dirty scripts I'm using right now, there's got to be a better way:

$ cat current/preprocess.sh
if [ $(basename -s .pdf "$1") = $(basename "$1") ]; then
    cp "$1" "$2"
else
    pdftotext -nopgbrk -fixed 6 "$1" - | awk -f $(dirname $(realpath "$0"))/preprocess.awk >"$2"
fi

$ cat current/preprocess.awk
BEGIN { OFS=","; FIELDWIDTHS = "9:9 4 1:29 1:15 1:15 1:12" }
/Date.*(Balance|Details)/,/BALANCE CARRIED FORWARD|Summary/ {
    gsub(",","", $0);
    BALANCE = $6;
    if (!(/Balance|Details/)) {
        if (!($1 ~ /^ *$/)) { DATE = $1 };
        if (!($2 ~ /^ *$/)) { TYPE = $2 };
        DESC = DESC " " $3;
        IN = $4;
        OUT = $5;
        if (!(IN ~ /^ *$/ && OUT ~ /^ *$/ && BALANCE ~ /^ *$/)) {
            match(DESC, /ADVICE CONFIRMS +([^ ]*)/, advice)
            ADVICE=advice[1]
            print DATE, TYPE, ADVICE, DESC, IN, OUT, BALANCE
            DESC = ""
        }
    }
}

cat trade/preprocess
#!/bin/sh
set -eu
T=$(mktemp)
ssconvert -S "$1" "${T}.csv"
cat ${T}.csv.* | grep GBP | grep -v Dividend | sed -e 's/GBP/£/' >"$2"

cat cc/preprocess
pdftotext -nopgbrk -fixed 4 "$1" - | awk -f $(dirname $(realpath "$0"))/preprocess.awk >"$2"

TMP=$(mktemp)
pdftotext -layout "$1" - | sed -re 's/ +/ /g' >"$TMP"
(
grep -E "DB Value Date:" "$TMP" | cut -f2 -d:|cut -f2 -d' '
grep -Eo "CR Value Date:.*" "$TMP" | cut -f2 -d:|cut -f2 -d' '
grep -C1 "Account Number" "$TMP" | tail -n1|sed -e 's/^ *//'
cat "$TMP" | grep "your account"|sed -re 's/^.*your account ([^ ]+).*$/\1/'
grep -E "Our Ref:" "$TMP" | cut -f2 -d: |sed -e 's/^ *//'
if grep "ROMANIA" "$TMP" >/dev/null; then
    echo ROMANIA
else
    grep -C1 "Senders Reference:" "$TMP" | tail -n1|sed -e 's/,/ /g' -e 's/^ *//'
fi
grep -C1 "Details of Payment" "$TMP" | tail -n1| sed -e 's/,/ /g'
if grep "ADVICE OF CREDIT" "$TMP" >/dev/null; then
    echo "CREDIT"
    grep -Eo "Amount Credit:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
    grep -Eo "Amount Received:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
elif grep "ADVICE OF DEBIT" "$TMP" >/dev/null; then
    echo "DEBIT"
    grep -Eo "Amount Debited:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
    grep -Eo "Original Amount:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
else
    echo "UNKNOWN TYPE" >&2
    exit 1
fi
grep -Eo "Charges:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
) | sed -e 's/GBP/£/g' -e 's/EUR/€/g'| paste -s -d, >"$2"
ER=$(grep -Eo "Exchange Rate:.*" "$TMP"|cut -f2 -d:|sed -e 's/^ *//')
if grep "ADVICE OF CREDIT" "$TMP" >/dev/null && grep "40XXXXXXXXXXXX" "$TMP" >/dev/null && grep Apr "$TMP" >/dev/null && [ -n "$ER" -a "$ER" != "1.00000000" ]; then
(
   # only have statements for 1 side of the transaction here between 2 accounts
   # have to synthesize the other side based on the exchange rate
    grep -E "DB Value Date:" "$TMP" | cut -f2 -d:|cut -f2 -d' '
    grep -Eo "CR Value Date:.*" "$TMP" | cut -f2 -d:|cut -f2 -d' '
    echo
    echo "40YYYYYYYYYYY DEBIT"
    grep -E "Our Ref:" "$TMP" | cut -f2 -d: |sed -e 's/^ *//'
    echo
    echo autogenerated based on exchange rate
    echo "DEBIT"
    AMOUNT=$(grep -Eo "Amount Credit:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:|sed -e 's/ *GBP *//')
    AMOUNT2=$(zsh -c "zmodload zsh/mathfunc; echo \$(( int($AMOUNT * $ER * 100.0 + 0.5) / 100.0 ))")
    echo "EUR $(printf '%.2f' $AMOUNT2)"
    echo "GBP $AMOUNT"
    grep -Eo "Charges:[^0-9]+[0-9.]+" "$TMP" | cut -f2- -d:
) | sed -e 's/GBP/£/g' -e 's/EUR/€/g' | paste -s -d, >>"$2"
fi
rm "$TMP"

Here is how some the rule file for the above preprocess one looks:

cat xxxx-advice.rules
fields date, _, dest, beneficiary, ref1, ref2, desc, typ, amount1, amount2, charges
date-format %d-%b-%y

amount %amount2

description %ref1 %ref2 %desc

if
CREDIT
    account2 assets:transfers:%ref1

if
40XXXXXXXXXXX.*CREDIT
    account1 assets:current-eur:edwin:xxxx

if
40YYYYYYYYYYYY.*CREDIT
    account1 assets:transfers:current:%ref1

if
40ZZZZZZZZZZZZ.*CREDIT
    account1 assets:transfers:loyisa:%ref1
    account2 assets:transfers:%ref1

if
ROMANIA
    account2 assets:current:edwin:romania

if
DEBIT
    account1 assets:transfers:%ref1

if
40XXXXXXXXXXX.*DEBIT
    account2 assets:current-eur:edwin:xxxx
    amount %amount2 @@ %amount1

Additional context

Move the Feature Reference out of our README, and into
https://hledger-flow.readthedocs.io/

Also update the docURL function to point to the new URL, and ensure that existing usages will generate valid links.

I've issued many hledger-flow import commands yesterday, and it could well be that one of them created include files where it shouldn't have. Now i'm unable to import.

~/Finance/import/personal/somebank$ hledger-flow import --sequential
Collecting input files...
Found 12 input files in 0.054998203s. Proceeding with import...
I couldn't find the right number of directories between "import" and the input file:
.../Finance/import/personal/somebank/depot/2011-include.journal

hledger-flow expects to find input files in this structure:
import/owner/bank/account/filestate/year/trxfile

Have a look at the documentation for a detailed explanation:
https://github.com/apauley/hledger-flow#input-files

I don't think it's worth it to figure out (just yet) how I got there.

But my question is: how do I safely remove all the include files generated by previous hledger-flow import runs, so that I may start fresh, from a known good state? Is the command below a good first approximation?

find . -type f | grep -E "[0-9]{4}-include.journal|all-years.journal" | xargs rm

On this page, @josephmturner and @adept have proposed to use file-specific rules, i.e. hyperspecific rules that would only apply to one specific imported file, typically because they only target one transaction or transactions from this specific timeframe. The main reason for this is to avoid cluttering the main rules file while still being able to correctly allocate specific transactions, even by overriding general rules in some cases. The second advantage is in terms of performance optimization, as these rules won't be unnecessarily evaluated on every single file.

I think it would make sense for hledger-flow to allow similar file-specific rules and to enforce a way to implement them. In your example directory, I'd see it as:

└── import
    └── gawie
        └── bogart
            └── cheque
                ├── 1-in
                │   └── 2016
                │       ├── 123456789_2016-03-30.csv
                │       └── ...
                ├── 2-preprocessed
                │   └── 2016
                │       ├── 123456789_2016-03-30.csv
                │       └── ...
                ├── 3-journal
                │   └── 2016
                │       ├── 123456789_2016-03-30.journal
                │       └── ...
                ├── bogart-cheque.rules                 # General rules
                ├── ...
                └── rules                               # <-- File-specific rules go here
                    └── 2016
                        └── 123456789_2016-03-30.rules  # File-specific rules, same name as file

Process

Now, as I don't think hledger allows to specify using multiple rules files, we'd have to include the general rules in the specific rules. I'd see the process as following (but maybe there's a better way to do it):

1. On hledger-flow import, check if file-specific rules exist and create them if they don't (don't overwrite) for all detected CSVs. Write the following in each new file:

   include ../../../../../bogart.rules  # if file exists
   include ../../bogart-cheque.rules    # if file exists
   include ../../../../../all.rules     # if file exists (just an idea, that's where I put my generic transactions account-assignment rules, feel free to ignore, as it can also be included manually in any of the above files)
   
   # Write specific transactions rules below this line
   

2. Then, to import, use the file-specific rules only (as they contain all generic rules).

Potential issues

• If there are no rules files whatsoever, there's nothing to include in the specific files, and hledger will complain on an empty rules file. But I suppose that an error message indicating that it cannot import without a rules file would make sense then.
• Maybe it could be difficult for some users to understand where to put their account-identification rules? Or it could incite avoiding generic rules and favoring transaction-specific rules (inefficient)? But I suppose hledger-flow users know better ;-)

Is your feature request related to a problem? Please describe.

Hi, this is a great project! This has helped me make full-fledged-hledger (and hledger!) significantly more accessible, thank you for that.

Being new to hledger/PTA and hledger-flow, I may just be overlooking how to do what I want to do with existing features. If that is the case, I apologize and would appreciate guidance on what to do.

There are a couple things I would like to do which may or may not be related, so I'll throw them out there together:

• hledger seems to support tags, and they seem useful (maybe not?). Can I use them with hledger-flow and csv imports? I tried defining the field in CSV import rules, but it did not seem to work out ok.
• I have a bunch of "projects" each with their own (virtual) accounts and expenses, liabilities, etc, should I keep them in my primary hledger-flow import or should I separate the projects in some fashion? Maybe you have other recommendations for these types of "virtual" accounts?
• I have some use-cases where an "invoice" or "bill" comes in, and then "multiple parties" contribute to those liabilities, though the actual payment of the invoice/bill is separate. I am having a difficult time understanding the clear/obvious way to track this in hledger-flow. For example, when the invoice is received, or a contribution is made, or the bill is actually paid, I am not sure what to put in my CSV for the accounts.
• If I have different expenses go to expenses:payables:Acme Corp and expenses:payables:Pizza Shop and such, but would also like to say those are associated with a particular project or tag in reports, should I be using tags or some other tricks with accounts?

Version and Runtime Information

hledger-flow 0.13.2.0

Add a --quiet switch which would suppress all output, except for failures causing the program to exit.

This is incompatible with --verbose, if both are specified we should exit with an error message.

I'd like to see some output when running hledger-makeitso import

Background: hledger-makeitso import repeatedly failed for me when I tried makeitso for the first time today. I eventually figured that I called my preprocess script preprocessor instead of preprocess

Not really a feature request, but not sure where to document this.

Below is a setup for GitHub Actions, to validate a repository with hledger-flow

# .github/workflows/hledger-flow.yml

name: Validate hledger-flow

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1

    - name: Install hledger
      run: docker pull dastapov/hledger

    - name: Install hledger-flow
      run: curl -L https://github.com/apauley/hledger-flow/releases/download/v0.12.4.0/hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027.tar.gz | tar xvz && mv hledger-flow_Linux_x86_64_v0.12.4.0_4b9b027/hledger-flow .

    - name: Grant permissions to create files
      run: chmod 777 -R ./my-finances

    - name: Test hledger file
      run: docker run --name="ledger" -v $(pwd):/data dastapov/hledger ./hledger-flow import ./my-finances

Hi, How do I import Y=20 as 2020?
When I use %Y it comes across as 0020

OS X
hledger-flow -v import

contents of rules file:
skip 1

fields date, description, amount
date-format %-m/%-d/%--Y
currency $

error code:
Exit code 1
Error output:
hledger: error: could not parse "6/7/20" as a date using date format "%-m/%-d/%--Y"

Describe the bug
I use .timeclock files in one of my time 'account' directories where there is a preprocess and a construct script.
The preprocess script is called with $1 = $filename.timeclock and $2 = $filename.csv. The problem is that my
preprocess script does not alter the format of the file, it just cleans it. I end up with a csv file that is still a timeclock
file, but has a .csv extension.

This triggers an error when in the construct script, i then make use of that file in an !include

$ cat construct 
#!/usr/bin/env bash

source common.sh

hledger print -f - <<EOF
$(<"${my_dir}/aliases.journal")
!include ${1}
EOF

when the construct script is run with $1 = $filename.csv

hledger-1.20.2: sorry, CSV files can't be included yet
caught ERR (exit code : 1)

This behaviour did not cause any error in the past, hledger would just figure out from the content file what format it's in.
Now the behaviour has changed, and hledger will complain.

I guess this falls under the discussion of should preprocess always result in a CSV file. Preprocess could well be
a cleanup step that does not change the extension of the file, but then how to signal this?

The workaround is for me to pack all the steps in a single construct file.

Reproduction

• Create a .timeclock file in 1-in
• Create a process script that only cats $1 to $2
• Create a construct script that feeds !include $1 to hledger print -f -
• Error: CSV files can't be included yet

Version and Runtime Information

RuntimeOptions {baseDir = "/home/lestephane/Vault/Finance/", importRunDir = "import/special/timespent/hours-in-project/", importStartYear = Nothing, onlyNewFiles = False, hfVersion = "hledger-flow 0.14.2.0 linux x86_64 ghc 8.10", hledgerInfo = HledgerInfo {hlPath = FilePath "/home/lestephane/Finance/hledger", hlVersion = "hledger 1.20.2"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,10], versionTags = []}}, verbose = True, showOptions = True, sequential = True}

Background
I have PDF statements from my bank which i convert using a construct script (combination of pdftotext and awk script).

During the hledger-flow import invocation, I see some error, and indeed the corresponding journal is never created.

Collecting input files...
Found 23 input files in 0.134708928s. Proceeding with import...
...
hledger: -:161:1:
    |
161 | 2018-02-30 "Abschluss lt. Anlage 1 PN:905"
    | ^^^^^^^^^^
well-formed but invalid date: 2018-2-30

2019-04-15 17:04:58.505274683 EEST	hledger-flow End:   executing 'girokonto/construct' on 'girokonto/1-in/2018/Kontoauszug_vom_28.02.2018_05.03.2018.pdf' ExitFailure 1 (0.082273261s)
...
Imported 23 journals in 2.203744436s

the problem is twofold:

• there is now a YYYY-include.journal file that contains the !include of a file that does not exist on the filesystem in 3-journal
• the importcommand did not return a non-zero exit code (despite the construct ExitFailure 1). So, one will only see a problem later (potentially days later), when loading the files with hledger. By then the user won't know what made his file disappear.

$ hledger-ui -f all-years.journal 
...
in file included from /Finance/import/personal/bbbank/2018-include.journal,
/Finance/import/personal/bbbank/girokonto/2018-include.journal:4:97:
  |
4 | !include 3-journal/2018/Kontoauszug_vom_28.02.2018_05.03.2018.journal
  |                                                                                                 ^
No existing files match pattern: 3-journal/2018/Kontoauszug_vom_28.02.2018_05.03.2018.journal

(in v0.11.1.2)

I need to define two account aliases for accounts that I now use in multiple preprocess and construct files, to shield myself from having to modify multiple locations if I ever need to rename those two accounts (a distinct possibility)

alias aws:personal = Personal:Expenses:Admin:Hosting:AWS
alias aws:businessone = BusinessOne:Expenses:Admin:Hosting:AWS

Problem: I don't see the option to create a journal file that is included by all journal files.

There is a all-years.journal at the top-level, but that file is generated.

Without support from hledger-flow for adding a global journal, I don't see how to achieve this cleanly.

I'm going to try to use a yearly pre-import that is shared in multiple year subdirectories using symlinks, and will report whether that is an acceptable workaround.

btw, I tried running in --verbose mode, but the log output does not report any attempt to look for manual pre / post import journal. Anyway, I will try, and will report.

$ hledger-flow import --verbose 2>&1 | grep -E "_manual_|pre-import|post-import"
(no output)

$ hledger-flow-v0.12.3.1 --version
hledger-flow 0.12.3.1 linux x86_64 ghc 8.6
$ hledger --version
hledger 1.15.2
$ hledger-flow-v0.12.3.1 report --sequential --verbose
Report generation is still a work-in-progress - please let me know how this can be more useful.

Keep an eye out for report-related pull requests and issues, and feel free to submit some of your own:
https://github.com/apauley/hledger-flow/pulls
https://github.com/apauley/hledger-flow/issues

2019-10-13 12:39:24.644119269 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2017 balance --pretty-tables --quarterly --flat --no-total transfer
2019-10-13 12:39:25.337676849 EEST	hledger-flow End:   hledger --file all-years.journal --period 2017 balance --pretty-tables --quarterly --flat --no-total transfer ExitSuccess (0.693479539s)
2019-10-13 12:39:25.338883089 EEST	hledger-flow Wrote reports/all/2017/transfer-balance.txt
2019-10-13 12:39:25.339109965 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2018 balance --pretty-tables --quarterly --flat --no-total transfer
2019-10-13 12:39:26.022556443 EEST	hledger-flow End:   hledger --file all-years.journal --period 2018 balance --pretty-tables --quarterly --flat --no-total transfer ExitSuccess (0.683417781s)
2019-10-13 12:39:26.023754435 EEST	hledger-flow Wrote reports/all/2018/transfer-balance.txt
2019-10-13 12:39:26.02401414 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2019 balance --pretty-tables --quarterly --flat --no-total transfer
2019-10-13 12:39:26.707729553 EEST	hledger-flow End:   hledger --file all-years.journal --period 2019 balance --pretty-tables --quarterly --flat --no-total transfer ExitSuccess (0.683678604s)
2019-10-13 12:39:26.708550757 EEST	hledger-flow Wrote reports/all/2019/transfer-balance.txt
2019-10-13 12:39:26.708752042 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2020 balance --pretty-tables --quarterly --flat --no-total transfer
2019-10-13 12:39:27.341601174 EEST	hledger-flow End:   hledger --file all-years.journal --period 2020 balance --pretty-tables --quarterly --flat --no-total transfer ExitSuccess (0.63282636s)
Did not write 'reports/all/2020/transfer-balance.txt' (hledger --file all-years.journal --period 2020 balance --pretty-tables --quarterly --flat --no-total transfer) ExitSuccess
2019-10-13 12:39:27.342117833 EEST	hledger-flow Begin: hledger --file all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value
hledger: Flag requires argument: --value (use -h to see usage)
2019-10-13 12:39:27.424352124 EEST	hledger-flow End:   hledger --file all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value ExitFailure 1 (0.082209168s)

=== Begin Error: hledger --file all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value ===
External command:
/home/lestephane/.local/bin/hledger --file /home/lestephane/[...]/all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value
Exit code 1
Error output:
hledger: Flag requires argument: --value (use -h to see usage)

=== End Error: hledger --file all-years.journal --period 2017 incomestatement --depth 2 --pretty-tables not:equity --cost --value ===

The doc says:

--value=TYPE[,COMM]  TYPE is cost, end, now or YYYY-MM-DD.
                          COMM is an optional commodity symbol.
                          Shows amounts converted to:
                          - cost using transaction prices, then optionally to
                          COMM using period-end market prices
                          - period-end market value, in default valuation
                          commodity or COMM
                          - current market value, in default valuation
                          commodity or COMM
                          - market value on the given date, in default
                          valuation commodity or COMM

so TYPE is no longer optional...

This issue is not really blocking me, but I wanted to look at what reports it generates, and start asking questions about how to generate my own (without having to modify the haskell source code obviously).

Describe the bug
When using hledger-flow on windows the preprocess and construct scripts that are called from src\CSVImports.hs (line 70/71) cannot be called.

On Windows executable files or scripts need a file-extension, not just only an executable attribute, hence windows doesn't know what to do with the preprocess and construct scripts.
You could for example try to find a preprocess.bat and construct.bat of preprocess.py and construct.py to get these scripts to execute.

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.13.2.0 mingw32 x86_64 ghc 8.8

This is the last git version I'm running (just updated)

Desktop (please complete the following information):

• OS: Windows10 Pro
• Version 1909

Add a new dependency to the project that can log debugging output to log files.

Change the verbose logging to rather use the logger.

This will probably impact #40 - will we then need multiple levels of verbose output to stderr if we have a logger?
Maybe we can add an option to the effect of --log-to-stderr instead of to a file.

Update: RIO has built in logging, consider using this.

Related to #14, #39 and #1

Allow multiple occurrences of the -v flag to indicate increased levels of verbosity.

Example:

$  hledger-flow report -vvv

In #5 we created an all-years.journal for each owner, bank and account.

Pointing your LEDGER_FILE to this all-years.journal may become slow for people with many years of input data.

Write a recent-years.journal next to each all-years.journal which contains only the 3 most recent years, where "3" should probably be a command-line option with a sensible default if not specified

Displaying the version of the command is currently implemented as a subcommand:

$ hledger-makeitso version
hledger-makeitso 0.10.0.0

Change this into a --version flag, with a short option of -V.

The short -V has to be capitalized because a lower-case -v is reserved for a verbose flag.

Background: I'm defining some rules files for financial institutions that other people may find useful as well (Revolut in EU and soon US, FFB in DE). I would like to share them in GitHub.

Problem: the rules files (.rules, preprocess) must be in the same directory where the yearly sub-directories are located, or in a parent directory. There is no way to 'discover' these files in some other sibling directory that has been fetched from another git repo.

I could use symlinks of course, except that I'm moving to another encryption driver for the statements directory, which does not support symbolic links yet.

So, there could be one external git repo with rules for many institutions, or one repo per institution. And then hledger-flow must know to look into it.

Maybe a HLEDGER_RULE_PATH environment variable?

$ cd GitRepos
$ git clone [email protected]:lestephane/hledger-flow-rules.git
$ tree hledger-flow-rules
hledger-flow-rules
├── ffb
│   └── ffb.rules
└── revolut
    ├── revolut.preprocess
    └── revolut.rules

$ cd ~/Finance
$ vim .envrc
...
path_add  HLEDGER_RULE_PATH  ~/GitRepos/hledger-flow-rules

$ tree Finance
import
├── personal
│   ├── ffb
│   │   └── depot
│   │       ├── 1-in
│   │       │   └── 2018
│   │       │       └── 2018.csv
│   ├── revolut
│   │   ├── checking
│   │   │   ├── 1-in
│   │   │   │   └── 2018
│   │   │   │       └── Revolut-EUR-Statement-2018.csv

I have 8 cores (4 physical + hyperthreading), and this is what it looks like when I run hledger-flow --import

Can we have a way to change the default? I may want to run only with 2 cores if there are a lot of open applications...

I would use those a lot...

Edit:
I am a new user and enter all data with hledger add (on the terminal).
At times I don't ask for a receipt because I can just memorize the price or it's just one item that I buy often.
~4 days later I remember, hey didn't I buy X last week? I think it was Thursday...

What I currently do:
hledger add
open external calendar see which date "last Thursday" is and copy the date into the hledger add prompt.

I'd like to do:
hledger add
Date [2019/04/23]: last thursday

I just realized that tab fills in the current date, so being able to type in 4 days ago isn't that urgent, but since yesterday works I expect other phrases to work and with tab completion as for yesterday l[ast] tab th[ursday] tab and 4 d[ays] tab a[go] tab would feel more convenient to me.

I have a question about how to use hledger-flow with this situation:

For some of my transactions, I would like to break them down into a group of transactions that are together. For example, if I shop at the hardware store and purchase some tools, and then some supplies for different projects, I'd like to be able to see the high-level transaction for all of those items, the taxes, etc, with the hardware store, but then also be able to see that $X was spent for project1 and $Y was spent for project2 while $Z was spent on tools.

AFAICT, this is possible to do in the PTA journal, but is there a way to do this with hledger-flow's csv imports?

If this is a new feature, I'd be happy to help contribute code to make it work, but would greatly appreciate some guidance / feedback on the best way to extend hledger-flow for this new feature.

Describe the bug

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ $HLEDGER_FLOW_EXECUTABLE --version
hledger-flow 0.15.0 linux x86_64 ghc 9.0 2b025fee477538a493cb7b5eaf163fd959aaec42

Is this the latest version?

Yes

Also include the runtime options of the command you ran, e.g:

/home/lestephane/.local/bin/hledger-flow-0.15.0 +RTS -N10 -RTS --show-options --batch-size 200 import
RuntimeOptions {baseDir = "/home/lestephane/Vault/Finance/", importRunDir = "import/personal/SCRUBBED/fondspolice/", importStartYear = Nothing, onlyNewFiles = False, hfVersion = "hledger-flow 0.15.0 linux x86_64 ghc 9.0 2b025fee477538a493cb7b5eaf163fd959aaec42", hledgerInfo = HledgerInfo {hlPath = "/home/lestephane/Vault/Finance/hledger", hlVersion = "hledger 5895a46, linux-x86_64"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [9,0], versionTags = []}, cores = 12, availableCores = 10}, verbose = False, showOptions = True, sequential = False, batchSize = 200}
Collecting input files from /home/lestephane/Vault/Finance/import/personal/SCRUBBED/fondspolice/
Found 1 input files in 0.000833711s. Proceeding with import...
Wrote include files for 1 journals in 0.002329759s
Imported 1/1 journals in 0.039248629s

To Reproduce

• Import a directory for which
  • There is a 1-in journal for year YYYY
  • There is no 1-in journal for year YYYY+1 << -- this part I'm not sure about
  • There is a _manual_/YYYY+2/post-import.journal

Expected behavior

• There should be a YYYY+2-include.journal
• The YYYY+2-include.journal should be included in all-years.journal

Desktop (please complete the following information):
N/A

Additional context
N/A

Describe the bug
The hledger executable is not found by hledger-flow on windows. It tries to search for the executable file hledger in the path, but on windows it should look for hledger.exe.

Changing the file it searches for in src\Common.hs on line 59 from maybeH <- which "hledger" to maybeH <- which "hledger.exe" fixes the problem on windows, so a platform check and saerch accordingly would be solution here.

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.13.2.0 mingw32 x86_64 ghc 8.8

This is the last git version I'm running (just updated)

Desktop (please complete the following information):

• OS: Windows10 Pro
• Version 1909

This example in the readme.org should have indentation.

2018-06-01 Savings Account Closing Balance
assets:Current:MyBank:Savings               $-234.56 = $0.00
equity:Closing Balances:MyBank:Savings

Should be indented like this:

2018-06-01 Savings Account Closing Balance
    assets:Current:MyBank:Savings               $-234.56 = $0.00
    equity:Closing Balances:MyBank:Savings

Describe the bug

I used to be able to go to a sub-account directory that had its preprocess construct 1-in etc
and to run hledger-flow import (without specifying an explicit directory), and it would just run the import in that directory. Now as of the latest release, it always imports everything, which takes on my machine 4 seconds for 2017+2018 across all my accounts concurrently. This will add up in time, especially since I'm not done importing all my years and accounts yet, and I use a lot of AWK & PDF conversion utilities in my preprocess and construct scripts. Everything gets re-run everytime.

If I attempt to work around the problem by running hledger-flow import . (that is, explicitly specifying the import directory) from withing the sub-account directory, I get an error (see reproduction below)

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.12.3.0 linux x86_64 ghc 8.6

Is this the latest version? YES

To Reproduce

• Go to the usual sub-account

~/Finance/import/personal/revolut/checking$ ls -1
1-in
2018-include.journal
3-journal
all-years.journal
construct
...

• Check that there are not many files there

~/Finance/import/personal/revolut/checking$ find . -name "*.journal" | wc -l
4

• Run the import command

~/Finance/import/personal/revolut/checking$ hledger-flow import 
Collecting input files...
Found 66 input files in 0.584593834s. Proceeding with import...
Imported 66 journals in 3.849160479s

• It looked for all files starting from ~/Finance/import (Unexpected)

• Attempting to workaround the problem by specifiying the import directory explicitly

~/Finance/import/personal/revolut/checking$ hledger-flow import .
hledger-flow: user error (
Unable to find an hledger-flow import directory at './' (or in any of its parent directories).

Have a look at the documentation for more information:
https://github.com/apauley/hledger-flow#getting-started
)

• It refuses to import (Unexpected)

Expected behavior

~/Finance/import/personal/revolut/checking$ hledger-flow-v0.11.1.2 import
Collecting input files...
Found 2 input files in 0.00324672s. Proceeding with import...
Imported 2 journals in 1.299441531s

A clear and concise description of what you expected to happen.

I'd much rather be able to import a single sub-account at a time. This is especially important when I work on my import scripts, when there is a lot of trial and error and I need a tighter feedback loop.

Describe the bug
When building hledger-flow in a Windows (Windows10) system, the executable is built correctly, but the tests fail, because of the path-separator. The included tests expect a unix path separtor / while the system returns the Windows path separator \.
This is only a thing in the tests; the code itself checks for the path-separator.

Installing and running the hledger-app and importing data works as expected

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.13.2.0 mingw32 x86_64 ghc 8.8

This is the last git version I'm running (just updated)

$ hledger-flow --show-options import
RuntimeOptions {baseDir = FilePath "C:\\Users\\max\\source\\hledger-flow-example\\", importRunDir = FilePath ".\\", useRunDir = False, hfVersion = "hledger-flow 0.13.2.0 mingw32 x86_64 ghc 8.8", hledgerInfo = HledgerInfo {hlPath = FilePath "C:\\Users\\max\\AppData\\Roaming\\local\\bin\\hledger.exe", hlVersion = "hledger 1.17.1.1"}, sysInfo = SystemInfo {os = "mingw32", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,8], versionTags = []}}, verbose = False, showOptions = True, sequential = False}
Collecting input files from C:\Users\max\source\hledger-flow-example\import\
Found 3 input files in 0.0079695s. Proceeding with import...
hledger-flow: C:\Users\max\source\hledger-flow-example\import\gawie\bogart\cheque\preprocess: createProcess: invalid argument (Exec format error)

To Reproduce
Can you reproduce your issue on these example files?
I was using the example files when running the --show-options import command above.

Steps to reproduce the behavior:

1. Download the code per instructions on the contributing guidelines
2. Then ran the command stack test --interleaved-output --pedantic
3. And got the following test error:

Registering library for hledger-flow-0.13.2.0..
hledger-flow> test (suite: hledger-flow-test)

### Failure in: 1:0
test\Common\Integration.hs:94
determineBaseDir produces an error message when given a non-existant dir
expected: Left "The provided directory does not exist: /path/to/dir"
 but got: Left "The provided directory does not exist: \\path\\to\\dir"
### Failure in: 2:7
test\CSVImport\Unit.hs:424
Convert a grouped map of paths, to a map with text contents for each file
expected: fromList [(FilePath ".\\import\\john\\bogartbank\\checking\\2018-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import/john/bogartbank/checking/3-journal/2018/2018-10-30.journal\n!include import/john/bogartbank/checking/3-journal/2018/2018-11-30.journal\n!include import/john/bogartbank/checking/3-journal/2018/2018-12-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\checking\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import/john/bogartbank/checking/3-journal/2019/2019-01-30.journal\n!include import/john/bogartbank/checking/3-journal/2019/2019-02-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\savings\\2017-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import/john/bogartbank/savings/3-journal/2017/2017-11-30.journal\n!include import/john/bogartbank/savings/3-journal/2017/2017-12-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\savings\\2018-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import/john/bogartbank/savings/3-journal/2018/2018-01-30.journal\n!include import/john/bogartbank/savings/3-journal/2018/2018-02-30.journal\n")]
 but got: fromList [(FilePath ".\\import\\john\\bogartbank\\checking\\2018-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import\\john\\bogartbank\\checking\\3-journal\\2018\\2018-10-30.journal\n!include import\\john\\bogartbank\\checking\\3-journal\\2018\\2018-11-30.journal\n!include import\\john\\bogartbank\\checking\\3-journal\\2018\\2018-12-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\checking\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import\\john\\bogartbank\\checking\\3-journal\\2019\\2019-01-30.journal\n!include import\\john\\bogartbank\\checking\\3-journal\\2019\\2019-02-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\savings\\2017-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import\\john\\bogartbank\\savings\\3-journal\\2017\\2017-11-30.journal\n!include import\\john\\bogartbank\\savings\\3-journal\\2017\\2017-12-30.journal\n"),(FilePath ".\\import\\john\\bogartbank\\savings\\2018-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include import\\john\\bogartbank\\savings\\3-journal\\2018\\2018-01-30.journal\n!include import\\john\\bogartbank\\savings\\3-journal\\2018\\2018-02-30.journal\n")]
### Failure in: 3:2
test\CSVImport\Integration.hs:95
All pre/post files on disk should be included
expected: fromList [(FilePath ".\\hlflow-1e1497e36b7682f9\\import\\john\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include _manual_/2019/pre-import.journal\n!include bank1/2019-include.journal\n!include bank2/2019-include.journal\n!include _manual_/2019/post-import.journal\n")]
 but got: fromList [(FilePath ".\\hlflow-1e1497e36b7682f9\\import\\john\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include _manual_\\2019\\pre-import.journal\n!include bank1\\2019-include.journal\n!include bank2\\2019-include.journal\n!include _manual_\\2019\\post-import.journal\n")]
### Failure in: 3:3
test\CSVImport\Integration.hs:119
All opening/closing files on disk should be included
expected: fromList [(FilePath ".\\hlflow-c14016fa77f9a37e\\import\\john\\bank1\\savings\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include 2019-opening.journal\n!include 3-journal/2019/2019-01-30.journal\n!include 2019-closing.journal\n")]
 but got: fromList [(FilePath ".\\hlflow-c14016fa77f9a37e\\import\\john\\bank1\\savings\\2019-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include 2019-opening.journal\n!include 3-journal\\2019\\2019-01-30.journal\n!include 2019-closing.journal\n")]
### Failure in: 3:4
test\CSVImport\Integration.hs:144
The price file should be included together with any pre/post files
expected: fromList [(FilePath ".\\hlflow-72de84936923ac78\\import\\2020-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include _manual_/2020/pre-import.journal\n!include john/2020-include.journal\n!include ../prices/2020/prices.journal\n!include _manual_/2020/post-import.journal\n")]
 but got: fromList [(FilePath ".\\hlflow-72de84936923ac78\\import\\2020-include.journal","### Generated by hledger-flow - DO NOT EDIT ###\n\n!include _manual_\\2020\\pre-import.journal\n!include john\\2020-include.journal\n!include ..\\prices\\2020\\prices.journal\n!include _manual_\\2020\\post-import.journal\n")]
### Failure in: 3:5
test\CSVImport\Integration.hs:198
John1: The include file contents should be the journal files
expected: "### Generated by hledger-flow - DO NOT EDIT ###\n\n!include 3-journal/2018/2018-10-30.journal\n!include 3-journal/2018/2018-11-30.journal\n!include 3-journal/2018/2018-12-30.journal\n"
 but got: "### Generated by hledger-flow - DO NOT EDIT ###\n\n!include 3-journal\\2018\\2018-10-30.journal\n!include 3-journal\\2018\\2018-11-30.journal\n!include 3-journal\\2018\\2018-12-30.journal\n"
Cases: 20  Tried: 20  Errors: 0  Failures: 6

hledger-flow> Test suite hledger-flow-test failed
Completed 2 action(s).
Test suite failure for package hledger-flow-0.13.2.0
    hledger-flow-test:  exited with: ExitFailure 1
Logs printed to console

Expected behavior
Pass the test

Desktop (please complete the following information):

• OS: Windows10 Pro
• Version 1909

Additional context
Add any other context about the problem here.

I've tried the verbose execution mode flag (-v) and the output looks really weird:

$ hledger-flow import -v
2019-04-13 08:57:52.983225837 EEST	hledger-flow Starting import
Collecting input files...
2019-04-13 08:57:53.091685212 EEST	hledger-flow Found 2 input files in 0.10839391s. Proceeding with import...
2019-04-13 08I: 5c7o:u5l3d.n0'9t1 7f5i3n7d7 4a nE EhSlTe	
... more garbage)

I cannot add the full output as, if later demangled, if could contain personal information.
If you need the full output, i can provide it to the author using send.firefox.com

Some notes about my setup:

• Ubuntu 16.04
• hledger-flow 0.11.0.0
• home directory is ecrypfs encrypted
• additionally, the directory i run the command in is encfs encrypted
• in all other respects, the command runs fine (with or without -v), just the output is screwed
• ghc 7.10.3
• stack version 1.9.3

To Reproduce

$ cd import
$ export PATH=".:$PATH"
$ ln -s ~/.local/bin/hledger-flow-async-batches-793f882bb22ac7b89a98077ee95b3464bbc5c0e0 hledger-flow
$ ln -s ~/.local/bin/hledger-32b42a6c8082e2567a73365d48696354f895c2f9 hledger
$ hledger-flow import
Unable to find hledger in your path. # <<- no time to investigate that one, it should have worked
You need to either install hledger, or add it to your PATH, or provide the path to an hledger executable.
$ which hledger
./hledger
$ hledger --version
hledger 1.21.99

Is your feature request related to a problem? Please describe.

I use autoposting rules to keep track of some out-of-pocket expenses that my company owes me.
If I make a change to an autoposting rule, I would like to review the ripple effect of the change, and of course version control it (not just the rule, all affected postings). If a change has many ripple effects, I'm much more likely to spot a problem at the gif diff stage. Currently I only see a problem when running hledger-ui after painfully looking for the transaction. If there is a problem with hledger itself (as when the synthetic transaction contains amounts that are formatted in an ambiguous way), I can't do anything, as the transaction does not 'exist' on the filesystem. This is how for example, a sum of 2.500 EUR from an autoposting gets interpreted as 2.5 EUR and I can't do anything about it, I can only specify a multiplier (*1, *-1, etc). If I could at least have these autopostings generated and visible in a file somewhere, i could apply an additional regex based search and replace in my build pipeline.

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.12.3.0 linux x86_64 ghc 8.6

Is this the latest version? YES

If your request includes commands you ran and the output, please also include
the runtime options with --show-options e.g:

Describe the solution you'd like

• one new file that is looked for by hledger-flow (besides construct and preprocess) -> rewrite-rules (name inspired from https://hledger.org/manual.html#re-write-rules-in-a-file's rewrite-rules.journal file parameter example)
• hledger-flow includes import works in all respect as before, except...
• ... if rewrite-rules is found, the journal generation command for 3-journal includes
  • the --auto command-line option
  • an additional -f command-line option pointing to the rewrite-rules
• rewrite-rules is not included in YYYY-include.journal, since the 3-journal files then already contain the side-effects of applying the rewriting rules.

(Note: I don't know how the 3-journal files are generated today, and I can't see any 3-journal related output in the verbose logs, so I indicated how I would do it using command-line options to the hledger print command)

Describe alternatives you've considered

The way I would do this today is by invoking

$ hledger-flow import
$ hledger print --auto > before.txt
(make a change to autoposting rules)
$ hledger print --auto > after.txt
diff before.txt after.txt

If the aggregated journal is rather large, it will become impractical to run a diff on it.

Potential problems
If rewrite rules are put in some other shared include file (à la #50) and not rewrite-rules files, hledger-flow might not detect that --auto needs to be added when generating the journals.

The behaviour is undefined in that case. The user is expected to put autopostings in a file by that name, which should standardize things. Since hledger-flow is opinionated anyway, that should be fine.

Is your feature request related to a problem? Please describe.

I would like to import CSV files, which use ; as a separatorchar

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.12.3.0 linux x86_64 ghc 8.6

Is this the latest version?

If your request includes commands you ran and the output, please also include
the runtime options with --show-options e.g:

$ hledger-flow --show-options import
 hledger-flow --show-options import
RuntimeOptions {baseDir = FilePath "/data/", hfVersion = "hledger-flow 0.12.3.0 linux x86_64 ghc 8.6", hledgerInfo = HledgerInfo {hlPath = FilePath "/usr/bin/hledger", hlVersion = "hledger 1.14.2"}, sysInfo = SystemInfo {os = "linux", arch = "x86_64", compilerName = "ghc", compilerVersion = Version {versionBranch = [8,6], versionTags = []}}, verbose = False, showOptions = True, sequential = False}

Our Example Statements Repository

https://gist.github.com/weitzj/228ea40f4db33bfe2f49d2d7e43a2871

Can you give examples of what you would like by running hledger-flow on these files?

I would like to have the same behaviour as:

hledger import --separator=";" --file=....

Even better would be that hledger-flow delegates all options to hledger import

Describe alternatives you've considered

Using hledger import --separator=";" and giving up on hledger-flow