Memory usage about fintech-to-ynab HOT 18 CLOSED

Are you using a free Heroku deploy?

I haven't updated my instance for a few weeks so I don't have any of the latest changes between you and Ross.

Also wondering if I should wait for Pull 6 before redeploying.

Thanks for all the work you guys are putting into this!

from fintech-to-ynab.

I am using free heroku to test this out yeah. I would hold off on deploying just now until we work out the memory issue.

#6 is just a README update. All the actual code has been merged now :)

from fintech-to-ynab.

@rienafairefr do you have any ideas on this? Do you think it will be in the library? I don't see anywhere in our code that will be causing a memory leak like this after only a few syncs.

from fintech-to-ynab.

FYI on docker it sits around 280mb for my dataset.

from fintech-to-ynab.

How many transactions do you have?

from fintech-to-ynab.

device_knowledge_of_server 520396382

from fintech-to-ynab.

I'm going to update my docker image. I'm pretty happy with this release now on docker.

from fintech-to-ynab.

I can get it setup on Dockerhub if you want so that it auto builds from master?

from fintech-to-ynab.

It's up to you. might make sense. I'll keep mine up to date, but point people to yours going forward.

from fintech-to-ynab.

from fintech-to-ynab.

@rossdargan it's consistently using around 5/600. I have around 3000 transactions in YNAB. However, even 100mb for that many transactions seems insane. That would be about 30kb per transaction, which they are 100% not?

from fintech-to-ynab.

@rienafairefr i will do some reading into Python profiling and see if i can work out where the memory usage is coming from

from fintech-to-ynab.

@scottrobertson Mines at 260mb now. Looks like profiling will be the way forward.

from fintech-to-ynab.

I got rid of gunicorn, it's dropped to ~400mb from higher 500's. I will keep an eye on it.

from fintech-to-ynab.

As a baseline, without any data in it a Client object gives around 30-40 MB memory usage, just tested. I've created an offline client and added 5000 transactions to it, memory usage goes up to only 55 MB. A test budget with around 800 total entities, only 40-50 MB. what's taking you to hundreds of MB, this is weird..

from fintech-to-ynab.

Interesting. I dont know much python, so it's very likely that I am doing something stupid in my code :D

from fintech-to-ynab.

Just so you know, in the most recent update 0.5.2 of the library I've added a possibility to do selective sync. Instead of using client.sync, you can do:

client.sync(['be_payees'])

It's still a beta test maybe it doesnt work very well, but normally with that then the client only contains the payees without the overhead of all the user data. This might help for this issue, and maybe the other one where sync takes too much time 👍

from fintech-to-ynab.

Oh nice :)

I think we will still do the full sync though each time as we want the latest transactions so we can take the category from them to auto fill that.

from fintech-to-ynab.