Script uses increasingly large amounts of memory on EC2 about metoffice_ec2 HOT 5 CLOSED

This seems to have gotten better a bit, but the underlying problem still seems to be there.

This is the memory usage of the container for the last three days. While the memory usage does grow way slower, it still is growing until it crashes at some point. Note that this is on v1.2.0 though, not latest.

I'll try to push the new version today and see whether anything changes.

from metoffice_ec2.

So it seems like we still see that issue on the latest version:

e089abeca5cc4c8684d29a0d8919a691: main: OutOfMemoryError: Container killed due to memory usage
f75d195e627843b58031f42f74a28f7a: main: OutOfMemoryError: Container killed due to memory usage

from metoffice_ec2.

Hmm... are we certain Python is using all the memory here? It's not, say, disk caching or something weird like that?

The only objects which persist in the ec2.py script are _LOG and a few constants.

If it is Python then a simple hack might be to change the ec2.py script to terminate (and hence flush its memory usage) when the SQS queue is empty; and then call ec2.py from a bash script which calls ec2.py from an endless while loop?!?

from metoffice_ec2.

Just had a quick chat with Flo:

• Analysing memory usage in containers isn't trivial.
• For now, let's just assume it is our Python script that's using all the memory.
• A quick-and-dirty hack is to wrap our Python script in a Bash script, as outlined in my comment immediately above.

from metoffice_ec2.

don't see this anymore, closing.

from metoffice_ec2.