syncforynab / fintech-to-ynab Goto Github PK
View Code? Open in Web Editor NEWAutomatically push Monzo and Starling transactions into YNAB in real time.
License: MIT License
Automatically push Monzo and Starling transactions into YNAB in real time.
License: MIT License
Once YNAB support passing in payee_name
, we can remove all our logic around payee lookups. This should happen mid next week.
Hey Scott.
I am struggling to get the Monzo auto import working.
I can run Import::Monzo.new(monzo_access_token, monzo_account_id, ynab_account_id).import
which works (from local and Heroku), the webhook is created Monzo side but I get nothing in YNAB when I complete transactions in Monzo. Cant seem to find logs anywhere either?
Any pointers? This code will be amazing if I can get it working! ๐
Hi Scott, and greetings from the bank2ynab
project. I've seen that you have a way to push transactions directly into YNAB, and it would be neat if we could incorporate that into our project, too. But I don't see a license in your project, so I am asking you explicitly for permission. (Also, maybe you should add a license?)
Kind regards from Austria!
Declined transactions still fire a webhook so the Monzo transactions are still being imported
Today I accidentally tried to pay for something with my Monzo card, but didn't have enough balance, so caused a failed transaction, as expected.
It did however still fire a webhook, hit my Heroku box, create an entry in YNAB, and mark it as cleared.
I'll try and re-simulate one with more logging, although it may just be that I'm on an outdated copy of this repo as I haven't re-deployed for a few weeks.
The healthcheck is currently broken for docker - I suspect because we will now be getting a 302 probably for /
This is how we currently do the check... but it's not great:
HEALTHCHECK CMD curl --fail http://localhost:5000/ || exit 1
Maybe we could add a proper healthcheck endpoint... If we knew we were getting errors with the sync it would be good to be able to report it here.
Since the previous issue I raised, the app has not been working for me at all.
Every command, even ping, fails with at=error code=H10 desc="App crashed"
.
Is noone else seeing this? I've even re-deployed to Heroku.
Any ideas?
Just logging this to make sure I don't forget about it
ERROR:Fintech to YNAB:Exception on /monzo [POST]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/src/main.py", line 47, in route_monzo
body, code = create_transaction_from_monzo(data['data'], settings, 0)
File "/usr/src/main.py", line 169, in create_transaction_from_monzo
entities_payee_id=entities_payee_id,
UnboundLocalError: local variable 'entities_payee_id' referenced before assignment
Some transactions are appearing in YNAB without a payee. Although the payee is being shown in in the logs
@scottrobertson really cool project :) will it work with barclays business bank accounts the import process? I'm really keen to match up. Plus how far back would the data be able to go? Could I for instance add into YNAB previous spending dating back few months?
We need a way for people to select the budget_id they want to use.
By default, we will just use the first budget. This will be fine for most people.
I am adding Starling support, so... we need a new name. Suggestions?
I'm not sure if this is something you can actually fix at your end but when I pay at the pump for fuel the price is always ยฃ1 for a day or so and then the transaction is adjusted to the actual amount.
This is not reflected in YNAB sadly since it doesn't seem to create another transaction so YNAB only shows ยฃ1 instead of ยฃ45 which the transaction actually was.
I got the following error on the first push to YNAB
2017-05-29T16:07:13.746830+00:00 heroku[web.1]: State changed from starting to up
2017-05-29T16:07:22.915289+00:00 heroku[router]: at=info method=POST path="/monzo?secret=[secret]" host=warm-brushlands-75902.herokuapp.com request_id=0b2a795e-7912-44eb-8272-4dc4448bd32e fwd="52.209.175.250" dyno=web.1 connect=0ms service=5537ms status=500 bytes=451 protocol=https
2017-05-29T16:07:22.911747+00:00 app[web.1]: Traceback (most recent call last):
2017-05-29T16:07:22.911728+00:00 app[web.1]: ERROR:Fintech to YNAB:Exception on /monzo [POST]
2017-05-29T16:07:22.911748+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
2017-05-29T16:07:22.911749+00:00 app[web.1]: response = self.full_dispatch_request()
2017-05-29T16:07:22.911749+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
2017-05-29T16:07:22.911750+00:00 app[web.1]: rv = self.handle_user_exception(e)
2017-05-29T16:07:22.911751+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
2017-05-29T16:07:22.911751+00:00 app[web.1]: reraise(exc_type, exc_value, tb)
2017-05-29T16:07:22.911752+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
2017-05-29T16:07:22.911753+00:00 app[web.1]: rv = self.dispatch_request()
2017-05-29T16:07:22.911753+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
2017-05-29T16:07:22.911754+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2017-05-29T16:07:22.911755+00:00 app[web.1]: File "python/main.py", line 47, in route_monzo
2017-05-29T16:07:22.911755+00:00 app[web.1]: body, code = create_transaction_from_monzo(data['data'], settings, 0)
2017-05-29T16:07:22.911756+00:00 app[web.1]: File "python/main.py", line 188, in create_transaction_from_monzo
2017-05-29T16:07:22.911756+00:00 app[web.1]: ynab_client.client.push(expected_delta)
2017-05-29T16:07:22.911757+00:00 app[web.1]: File "/app/.heroku/src/pynynab/pynYNAB/Client.py", line 143, in push
2017-05-29T16:07:22.911758+00:00 app[web.1]: raise WrongPushException(expected_delta, delta)
2017-05-29T16:07:22.911758+00:00 app[web.1]: WrongPushException
2017-05-29T16:08:25.807172+00:00 heroku[router]: at=info method=POST path="/monzo?secret=[secret]" host=warm-brushlands-75902.herokuapp.com request_id=f8459bcc-ac29-4d0a-a45c-5cd72fd76c1d fwd="52.48.43.86" dyno=web.1 connect=0ms service=2671ms status=200 bytes=201 protocol=https
ynab_client init() is called in the main function, and also here anytime the webhook is called,
https://github.com/scottrobertson/fintech-to-ynab/blob/master/python/routes.py#L39
Either we init and for every webhook call (then the main call is not needed), or we init only once when the app is created, in main (then the call for each route is not needed) ?
Could be interesting to set up a CI to run the tests on commit? Travis CI is free for open source projects.
There is a limit on the import ID, so we are going to have to shorten the prefix as it breaks when we have larger integers.
This will result in duplicates being created, but it needs to be done. Once things are a lot more solid, and the API is out of beta, we will tag 1.0.0, follow semver and stop any breaking changes happening as much as possible.
Hey @rienafairefr. After merging your branch, CI is failing
Write some
The following just happened:
This happened because we pushed the transaction into be_transactions, but then didnt end up syncing it with YNAB. Not sure what the best course of action is here.
*it's low at the moment as they have a huge bug in the mobile apps that were killing the DB.
This is pretty easy for the most part, except when there are multiple transactions of the same amount on the same day, from the same merchant.
We use the format for CSV's at the moment, which we could reuse
Fintech-To-YNAB:amount:date:occurrence
To get the occurrence
, we will need to fetch all transactions for that day (for that account), and work it out based on that. Given the limited scope of that call, it should be rather fast.
For current accounts you need a slightly different query to get your 'account_id' to then use for creating the webhook.
/accounts?account_type=uk_retail
So far webhooks from the current account are working as expected ๐
As part of the YNAB API release, I will be rewriting this library. In the process, i will be switching away from Python to using Ruby. The only reason it was in Python before was due to the nynab-api library being written in Python. I don't know much Python, so it's been a bit of a pain to maintain this. V1 will stay available for anyone who wants to continue to use the Python version.
I have created a project for this https://github.com/scottrobertson/fintech-to-ynab/projects/1
This ticket will be left open for discussions.
I'm getting this port bind error when deploying to Heroku.
Previously I was just updating a manual heroku deploy, but also getting this error when using the readme Heroku deploy button.
-- previous log stuff --
app[web.1]: DEBUG:requests.packages.urllib3.connectionpool:https://app.youneedabudget.com:443 "POST /api/v1/catalog HTTP/1.1" 200 None
app[web.1]: DEBUG:pynYNAB.ObjClient:server_knowledge_of_device 0
app[web.1]: DEBUG:pynYNAB.ObjClient:current_server_knowledge 6989
app[web.1]: DEBUG:pynYNAB.ObjClient:Server knowledge has gone up by 6989. We should be getting back some entities from the server
app[web.1]: DEBUG:pynYNAB.ObjClient:current_device_knowledge 0
app[web.1]: DEBUG:pynYNAB.ObjClient:device_knowledge_of_server 6989
app[web.1]: INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
app[web.1]: DEBUG:settings:assigning accounts and payees
heroku[web.1]: State changed from starting to crashed
heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
heroku[web.1]: Stopping process with SIGKILL
heroku[web.1]: Process exited with status 137
Disconnected from log stream. There may be events happening that you do not see here! Attempting to reconnect...
app[web.1]: DEBUG:pynYNAB.ObjClient:server_knowledge_of_device 0
app[web.1]: DEBUG:pynYNAB.ObjClient:current_server_knowledge 6989
app[web.1]: DEBUG:pynYNAB.ObjClient:Server knowledge has gone up by 6989. We should be getting back some entities from the server
app[web.1]: DEBUG:pynYNAB.ObjClient:current_device_knowledge 0
app[web.1]: DEBUG:pynYNAB.ObjClient:device_knowledge_of_server 6989
app[web.1]: INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
app[web.1]: DEBUG:settings:assigning accounts and payees
heroku[web.1]: State changed from starting to crashed
heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
heroku[web.1]: Stopping process with SIGKILL
heroku[web.1]: Process exited with status 137
-- continuously loops through server knowledge and crash --
@rossdargan the docker image is HUGE. Not sure why that python image is so big, some of the layers are > 150mb which is crazy. We need to look into an alpine based image at least.
We should allow background processing of webhooks. This will allow us to stop syncing on process boot, and also respond to webhooks much faster. This fixes our issues with timeouts.
We will use celery (or another if anyone has suggestions) + redis
I think we should move everything out of the flask code and have a separate monzo object. Currently it's really hard to test without spinning up flask. Ideally we should be able to just pass a JSON object into a class without needing the web server.
A post 1.0 thing though!
For larger accounts, we are facing timeouts. This is an issue on something like Heroku as the have a hard coded 30-second timeout.
I am going to look into how/if we can background these requests without bringing in another dependency such as redis for queuing.
To allow Starling to work easily, we need them to support Personal Webhooks. Righy now we need to create clients and I don't see them being approved very quickly/at all, which makes this project super difficult to use. It will apparently be worked on soon ๐
Import::Monzo.new(monzo_access_token, monzo_account_id, ynab_account_id).import
where am I meant to run this? through heroku console? or something else. Sorry as I know this will be a stupid question @scottrobertson
Trying to deploy to Heroku and it fails with:
/tmp/build_df733f9dfc8237247bc950768433ef0e/scottrobertson-fintech-to-ynab-44a78ef/requirements.txt (line 11))
Downloading functools32-3.2.3-2.zip
Complete output from command python setup.py egg_info:
This backport is for Python 2.7 only.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-x4eyvv8i/functools32/
! Push rejected, failed to compile Python app.
! Push failed
We should write up some instructions how to deploy this. At least how to deploy to Heroku for example.
I am having some issues getting this to work with my Monzo account.
when I try to run the import I am getting to following error:
NameError: undefined local variable or method
monzo_access_token' for main:Object
`
I am very likely missing something as the instructions are not very clear.
Any help would be great
Add a generic CSV importer
Similar to #46, we will select the first account, and allow them to override this if needed.
We need to support all the starling webhook types as we find them.
Just hit a "Memory quota exceeded" error on heroku. The app is using 592M of ram. That seems a bit excessive. Need to work out where this is happening, and if it's in our app, or in the library.
@rossdargan how do you feel about this? We ready to tag a "stable" release?
Waiting for YNAB to add support
Now that YNAB have payee management with auto categorising, i think we can get rid of all our logic:
Ive setup per instructions but i dont seem to get transactions added to ynab even though i verified the keys used and import works. Do pending transactions get sent immediately i make payment or do they have to clear? Is there any POST json i can use to test the ynab insert works?
I've set this up with Monzo -> YNAB to the best of my ability, using the Monzo and YNAB API respectively. I've set my YNAB_ACCESS_TOKEN
and my YNAB_ACCOUNT_ID
in the config vars, running on Heroku. It seems to be all set up correctly, but when I spend on my Monzo card nothing actually gets pushed to my YNAB account.
Am I missing something? Is there something I could share to help troubleshoot this? I've looked at the logs, but nothing other than bundler: failed to load command: puma (/app/vendor/bundle/ruby/2.4.0/bin/puma)
seems to look off to me. This error only fired once this morning when I started up the script.
I'm worried it will become really easy to scan for this web hook. It might be worth configuring the URL we listen too to include a configurable pretty shared secret.
Until ynab add a way to sign or encrypt the payload this might be the best we can do.
Hi,
I'm trying to get this to work with Monzo, but all my transactions are detected as duplicates. The only logging I see is:
2017-06-10T21:23:40.627213+00:00 app[web.1]: DEBUG:settings:A previous transaction for the payee Tesco has been found
2017-06-10T21:23:40.627384+00:00 app[web.1]: DEBUG:settings:A subcategory was not found for the previous transaction for Tesco
2017-06-10T21:23:40.627495+00:00 app[web.1]: DEBUG:settings:searching for payee Tesco
2017-06-10T21:23:40.627616+00:00 app[web.1]: DEBUG:settings:Creating transaction object
2017-06-10T21:23:40.632605+00:00 app[web.1]: DEBUG:settings:Duplicate detection
2017-06-10T21:23:40.637105+00:00 app[web.1]: DEBUG:settings:skipping due to duplicate transaction
I don't know if this is related to #19? Any ideas?
This could be used to set up and maintain scheduled payments in ynab :)
Post monzo feed items with YNAB category balance after webhooks.
Reading a discussion on calling init_session in pynYNAB repo here, I can reckon that in fintech-to-ynab there is no call to init_session, I'm dumbfounded why it seems to work then ? I'm missing something.
I updated to the YNAB API version this week and transactions have emoji and hashtags in the memo, where I had opted out of them in v1.
Just a heads up on what I'm working on adding now.
When we get a new transaction I now look for a previous transaction for this payee, and get the category set, and the payee actually used (some people tidy up the names of the payees).
Are you happy for me to sneak this in before 1.0?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.