Linux RSS infintely increasing though go heap is empty about go-python HOT 6 CLOSED

messageDict.Clear() will clear the content of the dict (ie: remove (eventually) its entries) but memory will still be retained by the backing structure of the dict.
you need to call PyObject.Decref() on it:

func main() {
    initPython()

    for {
        //Infinite loop, make some data and create python dict
        message := createMessage()
        messageDict := getPyDict(message)
        messageDict.Clear()
        messageDict.Decref()
    }
}

that's because PyDict_New() returns a new reference that should be owned by the calling code (and the calling code should take care of disposing it after use.)

see:
https://docs.python.org/2/c-api/intro.html#objects-types-and-reference-counts

hth,
-s

(feel free to reopen if something's astray)

from go-python.

Nope. Made no difference. Still the same, even after enabling Python GC manually via their API https://docs.python.org/2/library/gc.html (called gc.enable() function).

from go-python.

you're right. there are 2 other errors, leading to memory leaks:

• you need to call PyDict_Clear(messageDict) and not messageDict.Clear()
• you need to call pyKey.DecRef() and pyValue.DecRef() as PyDict_SetItem does not steal references

with these modifications, the following program:

package main

import python "github.com/sbinet/go-python"

func main() {
    initPython()

    for {
        //Infinite loop, make some data and create python dict
        message := createMessage()
        messageDict := getPyDict(message)
        python.PyDict_Clear(messageDict)
        messageDict.DecRef()
    }
}

func createMessage() map[string]interface{} {
    payload := make(map[string]interface{})
    payload["key_1"] = "value_1"
    payload["key_2"] = "value_2"
    payload["key_3"] = "value_3"
    return payload
}

func getPyDict(message map[string]interface{}) *python.PyObject {
    messageDict := python.PyDict_New()
    for key, value := range message {
        pyKey := python.PyString_FromString(key)
        pyValue := python.PyString_FromString(value.(string))
        python.PyDict_SetItem(messageDict, pyKey, pyValue)
        pyKey.DecRef()
        pyValue.DecRef()
    }

    return messageDict
}

func initPython() {
    //Init Python
    if err := python.Initialize(); err != nil {
        panic(err)
    }
}

stabilizes at 12.5m RSS on my 64b linux machine.

from go-python.

Cool. Works !!!. Thanks a lot, have been struggling with this for a while. I see that there is no corresponding function PyDict_Clear() available for Lists or Tuples. Does that mean that Clear() and subsequent DecRef() on the objects would be sufficient?

from go-python.

yes.

from go-python.

Noticed that Clear() infact calls DecRef() on the object after setting the pointer to null. So calling Clear() was enough for all other objects except for the dict where I had to make another extra call to PyDict_Clear().

from go-python.