@scottrobertson really cool project :) will it work with barclays business bank accounts the import process? I'm really keen to match up. Plus how far back would the data be able to go? Could I for instance add into YNAB previous spending dating back few months?

I've set this up with Monzo -> YNAB to the best of my ability, using the Monzo and YNAB API respectively. I've set my YNAB_ACCESS_TOKEN and my YNAB_ACCOUNT_ID in the config vars, running on Heroku. It seems to be all set up correctly, but when I spend on my Monzo card nothing actually gets pushed to my YNAB account.

Am I missing something? Is there something I could share to help troubleshoot this? I've looked at the logs, but nothing other than bundler: failed to load command: puma (/app/vendor/bundle/ruby/2.4.0/bin/puma) seems to look off to me. This error only fired once this morning when I started up the script.

Similar to #46, we will select the first account, and allow them to override this if needed.

Ive setup per instructions but i dont seem to get transactions added to ynab even though i verified the keys used and import works. Do pending transactions get sent immediately i make payment or do they have to clear? Is there any POST json i can use to test the ynab insert works?

I updated to the YNAB API version this week and transactions have emoji and hashtags in the memo, where I had opted out of them in v1.

@rossdargan the docker image is HUGE. Not sure why that python image is so big, some of the layers are > 150mb which is crazy. We need to look into an alpine based image at least.

The following just happened:

• Webhook came in
• YNAB raised Rate Limit error*
• Monzo sent the webhook again
• We returned "duplicate"

This happened because we pushed the transaction into be_transactions, but then didnt end up syncing it with YNAB. Not sure what the best course of action is here.

*it's low at the moment as they have a huge bug in the mobile apps that were killing the DB.

Could be interesting to set up a CI to run the tests on commit? Travis CI is free for open source projects.

Now that YNAB have payee management with auto categorising, i think we can get rid of all our logic:

http://docs.youneedabudget.com/article/611-payee-management

Reading a discussion on calling init_session in pynYNAB repo here, I can reckon that in fintech-to-ynab there is no call to init_session, I'm dumbfounded why it seems to work then ? I'm missing something.

I think we should move everything out of the flask code and have a separate monzo object. Currently it's really hard to test without spinning up flask. Ideally we should be able to just pass a JSON object into a class without needing the web server.

A post 1.0 thing though!

Just hit a "Memory quota exceeded" error on heroku. The app is using 592M of ram. That seems a bit excessive. Need to work out where this is happening, and if it's in our app, or in the library.

Hi Scott, and greetings from the bank2ynab project. I've seen that you have a way to push transactions directly into YNAB, and it would be neat if we could incorporate that into our project, too. But I don't see a license in your project, so I am asking you explicitly for permission. (Also, maybe you should add a license?)

Kind regards from Austria!

• Ben

I got the following error on the first push to YNAB

2017-05-29T16:07:13.746830+00:00 heroku[web.1]: State changed from starting to up
2017-05-29T16:07:22.915289+00:00 heroku[router]: at=info method=POST path="/monzo?secret=[secret]" host=warm-brushlands-75902.herokuapp.com request_id=0b2a795e-7912-44eb-8272-4dc4448bd32e fwd="52.209.175.250" dyno=web.1 connect=0ms service=5537ms status=500 bytes=451 protocol=https
2017-05-29T16:07:22.911747+00:00 app[web.1]: Traceback (most recent call last):
2017-05-29T16:07:22.911728+00:00 app[web.1]: ERROR:Fintech to YNAB:Exception on /monzo [POST]
2017-05-29T16:07:22.911748+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
2017-05-29T16:07:22.911749+00:00 app[web.1]:     response = self.full_dispatch_request()
2017-05-29T16:07:22.911749+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
2017-05-29T16:07:22.911750+00:00 app[web.1]:     rv = self.handle_user_exception(e)
2017-05-29T16:07:22.911751+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
2017-05-29T16:07:22.911751+00:00 app[web.1]:     reraise(exc_type, exc_value, tb)
2017-05-29T16:07:22.911752+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
2017-05-29T16:07:22.911753+00:00 app[web.1]:     rv = self.dispatch_request()
2017-05-29T16:07:22.911753+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
2017-05-29T16:07:22.911754+00:00 app[web.1]:     return self.view_functions[rule.endpoint](**req.view_args)
2017-05-29T16:07:22.911755+00:00 app[web.1]:   File "python/main.py", line 47, in route_monzo
2017-05-29T16:07:22.911755+00:00 app[web.1]:     body, code = create_transaction_from_monzo(data['data'], settings, 0)
2017-05-29T16:07:22.911756+00:00 app[web.1]:   File "python/main.py", line 188, in create_transaction_from_monzo
2017-05-29T16:07:22.911756+00:00 app[web.1]:     ynab_client.client.push(expected_delta)
2017-05-29T16:07:22.911757+00:00 app[web.1]:   File "/app/.heroku/src/pynynab/pynYNAB/Client.py", line 143, in push
2017-05-29T16:07:22.911758+00:00 app[web.1]:     raise WrongPushException(expected_delta, delta)
2017-05-29T16:07:22.911758+00:00 app[web.1]: WrongPushException
2017-05-29T16:08:25.807172+00:00 heroku[router]: at=info method=POST path="/monzo?secret=[secret]" host=warm-brushlands-75902.herokuapp.com request_id=f8459bcc-ac29-4d0a-a45c-5cd72fd76c1d fwd="52.48.43.86" dyno=web.1 connect=0ms service=2671ms status=200 bytes=201 protocol=https

This is pretty easy for the most part, except when there are multiple transactions of the same amount on the same day, from the same merchant.

We use the format for CSV's at the moment, which we could reuse

Fintech-To-YNAB:amount:date:occurrence

To get the occurrence, we will need to fetch all transactions for that day (for that account), and work it out based on that. Given the limited scope of that call, it should be rather fast.

We should write up some instructions how to deploy this. At least how to deploy to Heroku for example.

I'm getting this port bind error when deploying to Heroku.

Previously I was just updating a manual heroku deploy, but also getting this error when using the readme Heroku deploy button.

-- previous log stuff --
app[web.1]: DEBUG:requests.packages.urllib3.connectionpool:https://app.youneedabudget.com:443 "POST /api/v1/catalog HTTP/1.1" 200 None
app[web.1]: DEBUG:pynYNAB.ObjClient:server_knowledge_of_device 0
app[web.1]: DEBUG:pynYNAB.ObjClient:current_server_knowledge 6989
app[web.1]: DEBUG:pynYNAB.ObjClient:Server knowledge has gone up by 6989. We should be getting back some entities from the server
app[web.1]: DEBUG:pynYNAB.ObjClient:current_device_knowledge 0
app[web.1]: DEBUG:pynYNAB.ObjClient:device_knowledge_of_server 6989
app[web.1]: INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
app[web.1]: DEBUG:settings:assigning accounts and payees
heroku[web.1]: State changed from starting to crashed
heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
heroku[web.1]: Stopping process with SIGKILL
heroku[web.1]: Process exited with status 137
Disconnected from log stream. There may be events happening that you do not see here! Attempting to reconnect...
app[web.1]: DEBUG:pynYNAB.ObjClient:server_knowledge_of_device 0
app[web.1]: DEBUG:pynYNAB.ObjClient:current_server_knowledge 6989
app[web.1]: DEBUG:pynYNAB.ObjClient:Server knowledge has gone up by 6989. We should be getting back some entities from the server
app[web.1]: DEBUG:pynYNAB.ObjClient:current_device_knowledge 0
app[web.1]: DEBUG:pynYNAB.ObjClient:device_knowledge_of_server 6989
app[web.1]: INFO:werkzeug: * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
app[web.1]: DEBUG:settings:assigning accounts and payees
heroku[web.1]: State changed from starting to crashed
heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
heroku[web.1]: Stopping process with SIGKILL
heroku[web.1]: Process exited with status 137
-- continuously loops through server knowledge and crash --

Since the previous issue I raised, the app has not been working for me at all.
Every command, even ping, fails with at=error code=H10 desc="App crashed".

Is noone else seeing this? I've even re-deployed to Heroku.

Any ideas?

Today I accidentally tried to pay for something with my Monzo card, but didn't have enough balance, so caused a failed transaction, as expected.

It did however still fire a webhook, hit my Heroku box, create an entry in YNAB, and mark it as cleared.

I'll try and re-simulate one with more logging, although it may just be that I'm on an outdated copy of this repo as I haven't re-deployed for a few weeks.

Post monzo feed items with YNAB category balance after webhooks.

I am adding Starling support, so... we need a new name. Suggestions?

We should allow background processing of webhooks. This will allow us to stop syncing on process boot, and also respond to webhooks much faster. This fixes our issues with timeouts.

We will use celery (or another if anyone has suggestions) + redis

To allow Starling to work easily, we need them to support Personal Webhooks. Righy now we need to create clients and I don't see them being approved very quickly/at all, which makes this project super difficult to use. It will apparently be worked on soon 👍

https://trello.com/c/rviKbwNQ/47-personal-access-webhooks

Just a heads up on what I'm working on adding now.

When we get a new transaction I now look for a previous transaction for this payee, and get the category set, and the payee actually used (some people tidy up the names of the payees).

Are you happy for me to sneak this in before 1.0?

I am having some issues getting this to work with my Monzo account.

when I try to run the import I am getting to following error:

NameError: undefined local variable or method monzo_access_token' for main:Object
`
I am very likely missing something as the instructions are not very clear.

Any help would be great

Hi,

I'm trying to get this to work with Monzo, but all my transactions are detected as duplicates. The only logging I see is:

2017-06-10T21:23:40.627213+00:00 app[web.1]: DEBUG:settings:A previous transaction for the payee Tesco has been found
2017-06-10T21:23:40.627384+00:00 app[web.1]: DEBUG:settings:A subcategory was not found for the previous transaction for Tesco
2017-06-10T21:23:40.627495+00:00 app[web.1]: DEBUG:settings:searching for payee Tesco
2017-06-10T21:23:40.627616+00:00 app[web.1]: DEBUG:settings:Creating transaction object
2017-06-10T21:23:40.632605+00:00 app[web.1]: DEBUG:settings:Duplicate detection
2017-06-10T21:23:40.637105+00:00 app[web.1]: DEBUG:settings:skipping due to duplicate transaction

I don't know if this is related to #19? Any ideas?

For current accounts you need a slightly different query to get your 'account_id' to then use for creating the webhook.

/accounts?account_type=uk_retail

So far webhooks from the current account are working as expected 👍

This could be used to set up and maintain scheduled payments in ynab :)

@rossdargan how do you feel about this? We ready to tag a "stable" release?

Once YNAB support passing in payee_name, we can remove all our logic around payee lookups. This should happen mid next week.

Write some

There is a limit on the import ID, so we are going to have to shorten the prefix as it breaks when we have larger integers.

This will result in duplicates being created, but it needs to be done. Once things are a lot more solid, and the API is out of beta, we will tag 1.0.0, follow semver and stop any breaking changes happening as much as possible.

We need a way for people to select the budget_id they want to use.

By default, we will just use the first budget. This will be fine for most people.

The healthcheck is currently broken for docker - I suspect because we will now be getting a 302 probably for /

This is how we currently do the check... but it's not great:
HEALTHCHECK CMD curl --fail http://localhost:5000/ || exit 1

Maybe we could add a proper healthcheck endpoint... If we knew we were getting errors with the sync it would be good to be able to report it here.

Hey Scott.

I am struggling to get the Monzo auto import working.

I can run Import::Monzo.new(monzo_access_token, monzo_account_id, ynab_account_id).import which works (from local and Heroku), the webhook is created Monzo side but I get nothing in YNAB when I complete transactions in Monzo. Cant seem to find logs anywhere either?

Any pointers? This code will be amazing if I can get it working! 😀

I'm worried it will become really easy to scan for this web hook. It might be worth configuring the URL we listen too to include a configurable pretty shared secret.

Until ynab add a way to sign or encrypt the payload this might be the best we can do.

ynab_client init() is called in the main function, and also here anytime the webhook is called,

https://github.com/scottrobertson/fintech-to-ynab/blob/master/python/routes.py#L39

Either we init and for every webhook call (then the main call is not needed), or we init only once when the app is created, in main (then the call for each route is not needed) ?

Declined transactions still fire a webhook so the Monzo transactions are still being imported

Import::Monzo.new(monzo_access_token, monzo_account_id, ynab_account_id).import

where am I meant to run this? through heroku console? or something else. Sorry as I know this will be a stupid question @scottrobertson

We need to support all the starling webhook types as we find them.

Trying to deploy to Heroku and it fails with:

/tmp/build_df733f9dfc8237247bc950768433ef0e/scottrobertson-fintech-to-ynab-44a78ef/requirements.txt (line 11))
         Downloading functools32-3.2.3-2.zip
           Complete output from command python setup.py egg_info:
           This backport is for Python 2.7 only.
           
           ----------------------------------------
       Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-x4eyvv8i/functools32/
 !     Push rejected, failed to compile Python app.
 !     Push failed

Hey @rienafairefr. After merging your branch, CI is failing

https://circleci.com/gh/scottrobertson/fintech-to-ynab/32

Add a generic CSV importer

Some transactions are appearing in YNAB without a payee. Although the payee is being shown in in the logs

As part of the YNAB API release, I will be rewriting this library. In the process, i will be switching away from Python to using Ruby. The only reason it was in Python before was due to the nynab-api library being written in Python. I don't know much Python, so it's been a bit of a pain to maintain this. V1 will stay available for anyone who wants to continue to use the Python version.

I have created a project for this https://github.com/scottrobertson/fintech-to-ynab/projects/1

This ticket will be left open for discussions.

Waiting for YNAB to add support

• Add to Webhooks
• Add to Import

For larger accounts, we are facing timeouts. This is an issue on something like Heroku as the have a hard coded 30-second timeout.

I am going to look into how/if we can background these requests without bringing in another dependency such as redis for queuing.

I'm not sure if this is something you can actually fix at your end but when I pay at the pump for fuel the price is always £1 for a day or so and then the transaction is adjusted to the actual amount.

This is not reflected in YNAB sadly since it doesn't seem to create another transaction so YNAB only shows £1 instead of £45 which the transaction actually was.

Just logging this to make sure I don't forget about it

ERROR:Fintech to YNAB:Exception on /monzo [POST]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/src/main.py", line 47, in route_monzo
body, code = create_transaction_from_monzo(data['data'], settings, 0)
File "/usr/src/main.py", line 169, in create_transaction_from_monzo
entities_payee_id=entities_payee_id,
UnboundLocalError: local variable 'entities_payee_id' referenced before assignment