The effect of the command is printing of a formatted period to the terminal, however most commands result in printing some text. Something like get-many might suit better.

• set up sphinx stack (update setup.py and Makefile; also gitignore and README)
• generate documentation from docstrings (fix e.g. formatting issues if present)
• optionally add page about installation/first steps/etc.

It's a nice feature that each position can have a category. But what if we a add a statistics system in order to have a better overview of the certain categories.

For example:

And if we now use a statistics-print command this should print out how much money we pay on average for food and the total of food.

Because I'm using my own periods like october2019 I would also suggest adding a flag (taking an array as input) that keeps track of which periods should be calculated.

$ financeager statistics -p ['october2019', 'november2019']

TOTAL

Food  33,30€

AVERAGE:

Food: 16.65€

e.g. by extending the functionality of the --filter option, or by providing a new one.

WIP

• this is being solved by making financeager support Python plugins. It brings the opportunity to modularize the code base, and to enable extending the core package easily
• financeager version 0.23.0.0 was released, bringing plugin support

Current Situation

financeager can be used to communicate with a database on the local machine, or a database on a remote machine (which has a webservice installed).
Having both ways combined in a single package is powerful but has some smells. First of all it has to be reconsidered of how much value the 'remote' variant is at all.
When users want to use the 'local' variant,

• they still have to install dependencies relevant to the remote variant (requests, Flask)
• some package functionality is irrelevant (offline, httprequests, fflask, resources modules)

When users want to use the 'remote' variant,

• they have to configure financeager on the client side
• set up the webservice running behind an FCGI/WSGI

which might be too much of a hazzle already.

Now one major feature I'd like to introduce is a financeager client TUI (like e.g. tig or pudb). When used with the 'remote' variant however, the client would have to hold a copy of the remote database for performance/connectivity reasons, and sync this local copy periodically. This brings

• need for error-prone synchronization logic (re-inventing the wheel?)
• a local database which is already exactly what the local variant would do

Proposal

• the 'remote' variant is marked as deprecated
• the 'remote' variant implementation is split from the package
• the 'remote' variant implementation can be additionally installed and, by config, be selected for use
• the 'local' variant becomes the core of financeager

Then

• financeager can be extended by a TUI mode (see #50) which accesses the local database
• for synchronizing database files (if desired) users can
  • select a cloud storage service (e.g. Dropbox)
  • use version control, also in combination with a service, or a remote

This needs a requirements.txt for making it easier to collaborate on.

As mentioned in #11 (comment) , the tests are fragile and/or cross-platform incompatible.

On Windows 10 x64 I had 10 tests failing.
On MacOS Mojave I had 2 tests failing:

======================================================================
FAIL: test_invalid_host (test.test_httprequests.HttpRequestProxyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/abhibha/misc/financeager/test/test_httprequests.py", line 51, in test_invalid_host
    self.assertIn("Name or service not known", error_message)
AssertionError: 'Name or service not known' not found in "Error sending request: HTTPConnectionPool(host='weird.foodomain.nope', port=80): Max retries exceeded with url: /periods/2000/standard/1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x107d51e90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))"

Testing on error explanations is generally a bad idea. As seen here, for MacOS, the underlying error explanation is significantly different from most Linux ones: nodename nor servname provided, or not known.

Instead, test on the exact error type as much as possible. In this case that'd be NewConnectionError thrown by urllib3, which should be the same across platforms.

======================================================================
FAIL: test_verbose (test.test_cli.CliLocalServerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/mock.py", line 1209, in patched
    return func(*args, **keywargs)
  File "/Users/abhibha/misc/financeager/test/test_cli.py", line 150, in test_verbose
    "Loading custom config from {}".format(TEST_CONFIG_FILEPATH)))
AssertionError: False is not true

Classic case of new lines / whitespace breaking tests. The actual string printed to stderr (at-least on my Mac) is:

DEBUG 2019-10-15 18:58:48,266 financeager.config:48 Loading custom config from /tmp/financeager-test-config

Take note of the newline. string.endswith obviously fails. Instead, let's use in.

For user data directory and config directory, e.g. via appdirs. Replace hardcoded paths in __init__.py

In the current implementation, a single interface to tinydb exists: financeager.periods.TinyDbPeriod.
For adding different interfaces (e.g. for sqlite, Postgres), the base class should

• have abstract methods add_entry, remove_entry, update_entry, get_entry, get_entries
• provide the methods for validating and preprocessing input

In the clients.create() function, sinks and configuration are passed to construct a Client instance:

This due to both FlaskClient and LocalServerClient requiring these arguments.
If another custom Client subclass is implemented, it is possible that the instantiation above fails because Client.__init__() only requires the sinks argument.

Solution

• change signature to __init__(self, *, configuration, sinks)
• fix tests
• update docstrings
• update Changelog

Instead of python setup.py test; from a github actions run

Use

https://setuptools.pypa.io/en/latest/userguide/declarative_config.html
https://setuptools.pypa.io/en/latest/build_meta.html

from

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html#id7

to generate package structure that adheres to modern standards (avoid setup.py invocation, move meta-data)

Also https://snarky.ca/what-the-heck-is-pyproject-toml/

During 93dd26b three tests were disabled (previously not executed since not manually specified in test suite), since they are out-of-sync with the current state of the test_cli module and depend on manipulations at the direct interface to the webservice (instead of the command line interface which would already filter out the errors that are attempted to being triggered)

• proper highlighting of changes between versions
• when a new version is released, the additions to the changelog should also go into the GitHub release notes

E.g. using argcomplete

diff --git a/financeager/cli.py b/financeager/cli.py
index d972c0a..b0ac2d0 100644
--- a/financeager/cli.py
+++ b/financeager/cli.py
@@ -5,6 +5,8 @@ import argparse
 import os
 import sys
 
+import argcomplete
+
 from financeager import offline, communication, __version__,\
     init_logger, make_log_stream_handler_verbose, setup_log_file_handler
 import financeager
@@ -26,13 +28,14 @@ def main():
     directory are created. A FileHandler is added to the package logger.
     All command line arguments and options are parsed and passed to 'run()'.
     """
+    args = _parse_command()
     os.makedirs(financeager.DATA_DIR, exist_ok=True)
 
     # Adding the FileHandler here avoids cluttering the log during tests
     setup_log_file_handler()
 
     # Most runs return None which evaluates to return code 0
-    sys.exit(run(**_parse_command()))
+    sys.exit(run(**args))
 
 
 def run(command=None, config=None, verbose=False, **cl_kwargs):
@@ -251,6 +254,7 @@ least a frequency, start date and end date are optional. Default:
             subparser.add_argument(
                 "-p", "--period", help="name of period to modify or query")
 
+    argcomplete.autocomplete(parser)
     return vars(parser.parse_args(args=args))

• blocked by #13
• document procedure in README

• arose during 93dd26b
• ensure independency of tests (no clean up of period entries)
• alternative to use individual Flask app: use common Flask app but different period data bases per test method

Already deprecated in favor of fina

Currently a PeriodException is raised in the period would when validation fails OR when e.g. an entry cannot be found given an ID.
These should be distinguished in order to provide different feedback (validation fail translates to a 400 (bad request) while unavailable entry is a 404 (not found)).

Implementation idea:
Provide two distinct exception classes, and handle them separately in the server/resources module. It might be necessary to change the format of the response to e.g. {"type": "error", "error": "...error message or class..."}.

• functionality specific to cli, not to Client API
• requires wrapping logger.info for Out.info

When intending to add a recurrent entry, it easily happens that the -t recurrent option is forgotten:

fina add rent -100 -f monthly -s 01-01

This happily adds a standard entry at current date.

Proposal

When the --frequency option is present in the CLI, -t recurrent is assumed, and added if not present.

Consider adding a shortcut option -r/--recurrent.

https://github.com/nektos/act

ArgumentParser(description="") in cli module

See

This requires config values to be converted when using them.

financeager.period uses the schematics package to perform validation of data (i.e. standard and recurrent entries).
There are a couple of issues with the library:

1. When running tests (e.g. python -m test.test_period) a bunch of warnings is emitted about deprecated functionality of library (however the functionality complained about is used internally (?), not at the API level). See schematics/schematics#571.
2. The schematics project has not seen a release in over a year, and the activity on the repository is little.
3. There is a workaround of a schematics bug to perform validation at

   financeager/financeager/period.py

   Lines 154 to 158 in a19ee4f

   	# pass the kwargs twice because schematics API is inconsistent...
   	validation_model = ValidationModel(
   	raw_data=raw_data, **model_kwargs)
   	validation_model.validate(**model_kwargs)
   	return validation_model.to_primitive()

   .

It could be considered to replace the usage of schematics with e.g.

1. marshmallow. 4.2k stars, last release in Nov 2019, 141 contributors.
2. pydantic which does validation via type annotations (Python >= 3.6 only). 1.7k stars, last release in Nov 2019, 85 contributors
3. voluptuous. 1.5k stars, last release in Aug 2019, 61 contributors.
4. cerberus 1.9k stars, last release in Oct 2019, 52 contributors.
5. schema 1.9k stars, last release in Sep 2019, 50 contributors.
   It has to be estimated how much effort the migration would be (probably little since only to Schemas are used).

When running python setup.py test several warnings are shown at the end:

sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/financeager-qyuh0cmu/1909.json' mode='r+' encoding='UTF-8'>

This could be due to tearDownClass() in test_cli stopping the webservice without invoking closing the period database files (e.g. via Server.run('stop')).

For increased usability, a different user interface could be provided.

First of all this requires some architectural considerations, e.g. whether the frontend caches queries, and how it is synchronized with the backend (especially critical when using the webservice).

Frameworks for TUIs could be:

• urwid
• prompt_toolkit
• textual
  and for GUIs:
• PyQt5/PySide2
• remi (HTML based interface in webbrowser)
• dash (HTML/JS based interface in webbrowser)
• gooey (simply derives GUI from argparse parser)
• https://github.com/zauberzeug/nicegui

Assume some period databases exist in the data directory. When the flask app has been restarted, financeager list does not return anything. `Server.run('list') returns a list of names of TinyDbPeriod objects which have not yet been created (they are created when a request concerning a period is received).

Is this kind of 'lazy' loading desired?
Adding the content of the data directory to the list renders the response incorrect if TinyDbPeriod uses MemoryStorage.

• update all .format() strings to f-strings in the process using flynt
• don't ignore updates for pre-commit and gitlint dependencies no more

This is due to the implementation in listing.prettify() which does construct total-value entries for earnings and expenses when using stacked_layout = False (the default) but does not do so with stacked_layout = True, see

$ financeager print --stacked-layout
              Earnings               
Name               Value    Date  ID 
Unspecified          100.00          
  Food               100.00 10-12   1

-------------------------------------

              Expenses               
Name               Value    Date  ID 
Einkauf               50.00          
  Einkauf             50.00 10-12   2
Unspecified          500.00          
  Miete, October     500.00 10-12   1

$ financeager print
              Earnings                |               Expenses               
Name               Value    Date  ID  | Name               Value    Date  ID 
Unspecified          100.00           | Einkauf               50.00          
  Food               100.00 10-12   1 |   Einkauf             50.00 10-12   2
                                      | Unspecified          500.00          
                                      |   Miete, October     500.00 10-12   1
=============================================================================
Total                100.00           | Total                550.00          

Solution:
Construct total entries before branching into if stacked_layout: ..., and add them when constructing the stacked layout. Reuse them when constructing the side-by-side layout.

Related to #29.

Using the flask-backend, the command financeager list -f value=128 returns

Error handling request. Server returned 'Internal Server Error (500): unexpected error'

Possible problem: type-incompatibility between given string to look for, and stored numbers in db.

The current 0.X is too simple since it does not provide information whether a new version brings a breaking change, a feature, or a bugfix.

Proposal

Use a versioning scheme 0.major.minor.patch (cf. pipx) which works like the know semantic versioning major.minor.patch (until 1.0.0 is released).

• CategoryEntry.BASE_ENTRY_SORT_KEY, Listing.CATEGORY_ENTRY_SORT_KEY and CategoryEntry.DEFAULT_NAME are global class variables that have to be set at a correct point to have the desired effect on the formatted output of a 'print' request. This currently happens in the communication module
• during work on 93dd26b, it was noticed that test_cli.CliNoConfigTestCase.test_print modifies the global class variable CategoryEntry.DEFAULT_NAME which has an undesired effect on test_listing.PrettifyListingsTestCase.test_prettify (because of the way unittest invokes the tests?)

Solution:

• remove global class variables
• pass settings as arguments to listing.prettify, and down to formatting methods of the Entry classes

The print command should display a "remaining" field below the total income and expenses.

              Earnings                |               Expenses               
Name               Value    Date  ID  | Name               Value    Date  ID 
=============================================================================
Total              000.00             | Total              000.00          
Remaining          0.00

Currently, for every year a new database ('period') is created.

This has some shortcomings:

• by default, databases are distinguished by the year only, not by topic/project. However projects might span more time than one year, and it still could be desired to have a single database
• recurrent entries have to be copied each year (e.g. for rent) to the new period
• analyzing database content of multiple years by e.g. filtering using the list command is impossible but helpful

Proposal:

• Change terminology. Instead of 'period', use 'database' or the less technical 'record' (or 'project') 'pocket'. Reasoning: short CLI option remains unchanged (-p); term suits well into financing background
• Change backend structure. By default, the backend only holds one 'main' database (of unlimited time). The client can add different databases (for e.g. different projects), and has to specify the target database (unless default) whenever issuing a command.
• Provide migration functionality. The existing databases (i.e. per-year periods) have to be updated (change date fields from MM-DD to YYYY-MM-DD format) and combined. A migration command or script should be provided.
  • migration command in CLI: convert-periods-to-pocket
  • rename 'elements' in pocket module
  • CLI
    • support YYYY-MM-DD and MM-DD formats
    • deprecate copy command

The current implementation is not ideal, as described in a233f33.
Instead of the try-except block in _recover_data, Client.safely_run should be called, and the returned success flag be used.

financeager stores add/update/rm commands in case the request did not work out. This is intended to store commands when e.g. no internet connections is available. But storing is also triggered if an invalid command is issued, e.g. removing a non-existing ID (financeager rm -123). It is attempted to recover the invalid command the next time any command is run, and obviously it will fail again and never leaver the offline storage.
Ideas:

• offline storage only after server-communication timeout triggered
• remove offline storage entirely (I mean, we all know how to navigate the shell history...)

Install acc. to https://github.com/pypa/setuptools_scm
Slightly related to #41

This enables patching it in the test_cli module by simple custom implementation instead of using unittest.mock.

Waiting for data_dir to be evaluated in fflask.create_app makes any log messages from the Flask app start go lost when the app is not run in a terminal, but e.g. wrapped in a FCGI script.

Proposal: setup_log_file_handler should use a hardcoded path, and be invoked very soon when creating the app.

EDIT: The messages I was referring to are only emitted when starting the app from a script like

from financeager import fflask
app = fflask.create_app()
app.run()

The messages are

* Serving Flask app "financeager.fflask" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Obviously they are not relevant or even existing when wrapping the app for WSGI or FCGI.

Still, the file handler should be set up early to record all warnings (e.g. about data_dir not being specified).

It's not holding a config(uration) but rather the path where the custom config file resides.
Replace by e.g. config_path, and update the corresponding long command line option to --config-path.

Problem: financeager print --filters date=07- has no effect. Prints the entire period, would expect to get July entries only.
Possible solution: Rework the request parsing in resources.py.

From a personal usecase: The current month is usually the only thing I care about when running list. For this I use a date-filter

financeager list -f date=10-

for listing all entries of October.
For more convenience, a new option should be introduced:

financeager list --month 10
financeager list --month Oct
financeager list --month October
financeager list --month  # default to current month