Memory usage about facex HOT 12 CLOSED

Do you mean the model or training data?

from facex.

Yes, you're right, I meant the model. After loading the model into memory, the process uses ~600MB.

from facex.

Er, could you try it again? It shouldn't consume so much memory. On my computer, the demo consume about 47MB. Here is a screenshot:

Before running the program

After running the program

from facex.

I just double-checked your program and saw the same results. But in mine, the memory usage was extremely high. It only seems to happen when FaceX is called in a certain way.

I added a simple example here that causes the problem:
https://github.com/matthill/FaceX/tree/high_memory

from facex.

Does this happen when the model is being loaded or after being loaded?

In my computer, when loading the model, the memory consumption is high, but will decrease after loading. If this is what you come across, it's because of opencv. I use opencv format to store the model since it's easy to use. However, the loading process will consume much memory (will decrease after loading)

You can define your own binary format to solve this problem.

from facex.

Yes, it uses the memory when it first loads (which isn't a problem, as long as it's released). But when it's instantiated in another object (e.g., facewrapper in the branch I created) the memory is never released.

So in your main function, when you do this:
FaceX face_x(kModelFileName);

The memory is released.

But when you do this
facewrapper wrapper(kModelFileName);

The memory stays resident.

from facex.

This is what I'm seeing on this branch:
https://github.com/matthill/FaceX/tree/high_memory

from facex.

I tried your program. On windows, it just costs about 100M. On Linux, it indeed costs about 600M. I did some experiments and guess this is because there is a cache in OpenCV persistence module (just a guess). On Linux, the cache will retain for a long time.

Therefore, the solution is create a custom binary format to replace OpenCV persistence module I used.

from facex.

Are you sure that's the issue?

If I replace this section in regressor.cpp:
for( ; it != it_end; ++it, idx++ )
{
Fern f;
*it >> f;
ferns_.push_back(f);
}

With this one:
for( ; it != it_end; ++it, idx++ )
{
Fern f;
*it >> f;
//ferns_.push_back(f);
}

The memory issue seems to go away. The use of OpenCV's FileStorage seems exactly the same in both cases.

from facex.

I checked it for a long time, and still couldn't find where the problem lies. Therefore I asked on stackoverflow (http://stackoverflow.com/questions/31611741/how-to-debug-high-memory-consumption). Someone suggested me to use tcmalloc to check where the memory allocation happens. I followed his suggestion.

Then, mysterious thing happened: the problem was gone!

I still don't know where is the problem, but it seems use tcmalloc can solve it (just install gperftools and link the program with -ltcmalloc)

from facex.

just to report, running this on an old win32 box, there's only a single 350mb spike on startup, then falls down to <50mb while running. (not that bad, i'd say.)

from facex.

Yes, this problem doesn't happen on Windows.

from facex.