Code Monkey home page Code Monkey logo

Comments (8)

jpalus avatar jpalus commented on June 3, 2024 1

I don't think asan would show anything different than massif already does.

The reason why I used valgrind's massif in the first place is because valgrind's memcheck does not show any leaks so my assumption was the memory gets ultimately freed but only at exit. That's why I'm monitoring runtime memory use and not memory that was not freed at exit.

Anyway I found the issue:

neomutt/mx.c

Lines 1245 to 1249 in fa83255

const int grow = 25;
// Sanity checks
req_size = MAX(req_size, m->email_max);
req_size = ROUND_UP(req_size + 1, grow);

This logic is wrong as far as I can tell. Calculated rounded req_size is assigned to m->email_max which will always be selected in above MAX() and further increased by 25 due to increment and it will only keep on growing by 25 with each call. Since it is called per each email and every call allocates 25 more of sizeof(void*) + sizeof(int) which on my machine happens to be 12b, with 108260 emails that I have in big mailbox gives 108260*25*12=32478000. The 30MB leak that I'm seeing.

from neomutt.

flatcap avatar flatcap commented on June 3, 2024 1

Damn! You're good!

Thanks for the debugging.
I'll create a fix after I've done my chores -- lots of things need merging.

from neomutt.

flatcap avatar flatcap commented on June 3, 2024

I not sure that NeoMutt is leaking, but it does seem to be using quite a lot of memory.

When you open NeoMutt, obviously it allocates memory for all the mailboxes in the sidebar and all the emails in the open mailbox.
Even mailboxes that aren't open take up memory.

I did quick test whilst watching the memory in top (low-tech, I know).

I created two macros:

macro index <f5> "<change-folder>-<enter>"
macro index <f6> "<f5><f5><f5><f5><f5><f5><f5><f5><f5><f5>"

Then, I opened a small folder, then opened a very large folder.
Changing folder to - means open the previous folder.

I hit <f6> five times and waited.
That should open each folder 25 times.

The memory usage fluctuated as folders were opened and closed, but crucially it didn't creep upwards.
At least, not by any obvious amount.

I'll leave this open and keep thinking

from neomutt.

jpalus avatar jpalus commented on June 3, 2024

Maybe some more data to compare:

Memory use after opening initial small mailbox only without opening big one is negligible and less than 2MB.

Memory use after opening small mx -> big mx -> small mx

99.45% (34,998,419B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->92.71% (32,624,676B) 0x518404: mutt_mem_realloc (memory.c:131)
| ->61.58% (21,669,400B) 0x43033F: mx_alloc_memory (mx.c:1260)
| | ->61.58% (21,669,400B) 0x4B6E46: maildir_move_to_mailbox (shared.c:95)
| |   ->61.58% (21,669,400B) 0x4B1E40: maildir_read_dir (maildir.c:731)
| |     ->61.58% (21,669,400B) 0x4B1FEA: maildir_mbox_open (maildir.c:1127)
| |     | ->61.58% (21,669,400B) 0x431563: mx_mbox_open (mx.c:388)
| |     |   ->61.58% (21,669,400B) 0x43BCAC: change_folder_mailbox (dlg_index.c:696)
| |     |   | ->61.58% (21,669,400B) 0x44035F: op_main_change_folder (functions.c:1156)
| |     |   |   ->61.58% (21,669,400B) 0x4428F3: index_function_dispatcher (functions.c:3323)
| |     |   |     ->61.58% (21,669,400B) 0x43C782: dlg_index (dlg_index.c:1343)
| |     |   |       ->61.58% (21,669,400B) 0x40C7B6: main (main.c:1392)
| |     |   |         
| |     |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |     |   
| |     ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |     
| ->30.79% (10,834,700B) 0x430350: mx_alloc_memory (mx.c:1261)
| | ->30.79% (10,834,700B) 0x4B6E46: maildir_move_to_mailbox (shared.c:95)
| |   ->30.79% (10,834,700B) 0x4B1E40: maildir_read_dir (maildir.c:731)
| |     ->30.79% (10,834,700B) 0x4B1FEA: maildir_mbox_open (maildir.c:1127)
| |     | ->30.79% (10,834,700B) 0x431563: mx_mbox_open (mx.c:388)
| |     |   ->30.79% (10,834,700B) 0x43BCAC: change_folder_mailbox (dlg_index.c:696)
| |     |   | ->30.79% (10,834,700B) 0x44035F: op_main_change_folder (functions.c:1156)
| |     |   |   ->30.79% (10,834,700B) 0x4428F3: index_function_dispatcher (functions.c:3323)
| |     |   |     ->30.79% (10,834,700B) 0x43C782: dlg_index (dlg_index.c:1343)
| |     |   |       ->30.79% (10,834,700B) 0x40C7B6: main (main.c:1392)
| |     |   |         
| |     |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |     |   
| |     ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |     
| ->00.34% (120,576B) in 1+ places, all below ms_print's threshold (01.00%)

Memory use after opening small mx -> big mx -> big mx -> small mx:

99.71% (67,434,423B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->96.26% (65,101,648B) 0x518404: mutt_mem_realloc (memory.c:131)
| ->64.05% (43,320,800B) 0x43033F: mx_alloc_memory (mx.c:1260)
| | ->64.05% (43,320,800B) 0x4B6E46: maildir_move_to_mailbox (shared.c:95)
| |   ->64.05% (43,320,800B) 0x4B1E40: maildir_read_dir (maildir.c:731)
| |     ->64.05% (43,320,800B) 0x4B1FEA: maildir_mbox_open (maildir.c:1127)
| |     | ->64.05% (43,320,800B) 0x431563: mx_mbox_open (mx.c:388)
| |     |   ->64.05% (43,320,800B) 0x43BCAC: change_folder_mailbox (dlg_index.c:696)
| |     |   | ->64.05% (43,320,800B) 0x44035F: op_main_change_folder (functions.c:1156)
| |     |   |   ->64.05% (43,320,800B) 0x4428F3: index_function_dispatcher (functions.c:3323)
| |     |   |     ->64.05% (43,320,800B) 0x43C782: dlg_index (dlg_index.c:1343)
| |     |   |       ->64.05% (43,320,800B) 0x40C7B6: main (main.c:1392)
| |     |   |         
| |     |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |     |   
| |     ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |     
| ->32.03% (21,660,400B) 0x430350: mx_alloc_memory (mx.c:1261)
| | ->32.03% (21,660,400B) 0x4B6E46: maildir_move_to_mailbox (shared.c:95)
| |   ->32.03% (21,660,400B) 0x4B1E40: maildir_read_dir (maildir.c:731)
| |     ->32.03% (21,660,400B) 0x4B1FEA: maildir_mbox_open (maildir.c:1127)
| |     | ->32.03% (21,660,400B) 0x431563: mx_mbox_open (mx.c:388)
| |     |   ->32.03% (21,660,400B) 0x43BCAC: change_folder_mailbox (dlg_index.c:696)
| |     |   | ->32.03% (21,660,400B) 0x44035F: op_main_change_folder (functions.c:1156)
| |     |   |   ->32.03% (21,660,400B) 0x4428F3: index_function_dispatcher (functions.c:3323)
| |     |   |     ->32.03% (21,660,400B) 0x43C782: dlg_index (dlg_index.c:1343)
| |     |   |       ->32.03% (21,660,400B) 0x40C7B6: main (main.c:1392)

So it's seems pretty linear with 30MB allocated more per big mx open and matches pretty well allocation of 157MB after opening big mx 5 times that was shown in initial comment.

from neomutt.

flatcap avatar flatcap commented on June 3, 2024

OK, thanks.

If you've got the time, please could you build with the Address Sanitizer
It tracks all memory allocations / frees.

If there are any leaks, it prints out a report on exit telling us exactly where the allocations occurred.

from neomutt.

jpalus avatar jpalus commented on June 3, 2024

I guess the growing should occur only if original req_size (as present in function invocation) is higher than m->email_max. Not always.

from neomutt.

jpalus avatar jpalus commented on June 3, 2024

Made an attempt to fix the issue in #4089.

Memory use in situation from description (smal mx -> 5x big mx -> small mx)

  • before fix: ~157MB
  • after fix: 3.5MB

Although use when having big mx open is possibly still concerning: 500MB. To be analyzed independently though.

I will create one more PR for reducing allocation overhead a little.

from neomutt.

flatcap avatar flatcap commented on June 3, 2024

Thanks for diving in; I'll have a play in a second.

when having big mx open is possibly still concerning: 500MB

Long ago, I had an idea to create a :memory page, like firefox's about:memory
I'd be very interested in knowing where it's all being used.

from neomutt.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.