Comments (9)
mohit84
commented on May 18, 2024
It seems you are getting crash for CLI process, can you please attach a coredump with gdb and share the command output "thread apply all bt full" to debug it more?
from glusterfs.
wonderful1984
commented on May 18, 2024
(No debugging symbols found in /usr/local/sbin/gluster)
(gdb) set pagination 0
(gdb) set logging on
Copying output to gdb.txt.
Copying debug output to gdb.txt.
(gdb) thread apply all bt
(gdb) run
Starting program: /usr/local/sbin/gluster
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.34-112.oe2203sp1.x86_64
Missing separate debuginfo for /lib64/libtcmalloc_minimal.so.4
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/e5/0805e57e7c53c2a57576a2643850ea6e9d5a4a.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
[New Thread 0x155553253640 (LWP 294594)]
[Detaching after vfork from child process 294596]
[New Thread 0x155553052640 (LWP 294617)]
Welcome to gluster prompt, type 'help' to see the available commands.
[New Thread 0x155552e51640 (LWP 294618)]
[New Thread 0x155552c50640 (LWP 294619)]
gluster> volume status all clients
Thread 5 "glfs_epoll001" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x155552c50640 (LWP 294619)]
0x0000155555473744 in iobuf_get2 () from /usr/local/lib/libglusterfs.so.0
Missing separate debuginfos, use: dnf debuginfo-install e2fsprogs-1.46.4-17.oe2203sp1.x86_64 keyutils-libs-1.6.3-4.oe2203sp1.x86_64 krb5-libs-1.19.2-6.oe2203sp1.x86_64 libgcc-10.3.1-20.oe2203sp1.x86_64 libselinux-3.3-2.oe2203sp1.x86_64 libstdc++-10.3.1-20.oe2203sp1.x86_64 libtirpc-1.3.2-3.oe2203sp1.x86_64 libuuid-2.37.2-13.oe2203sp1.x86_64 libxml2-2.9.14-5.oe2203sp1.x86_64 ncurses-libs-6.3-5.oe2203sp1.x86_64 openssl-libs-1.1.1m-20.oe2203sp1.x86_64 pcre2-10.39-6.oe2203sp1.x86_64 readline-8.1-2.oe2203sp1.x86_64 userspace-rcu-0.13.0-5.oe2203sp1.x86_64 xz-libs-5.2.5-3.oe2203sp1.x86_64 zlib-1.2.11-22.oe2203sp1.x86_64
(gdb) bt
#0 0x0000155555473744 in iobuf_get2 () from /usr/local/lib/libglusterfs.so.0
#1 0x00001555532fb49c in socket_event_handler () from /usr/local/lib/glusterfs/11.1/rpc-transport/socket.so
#2 0x00001555554a836e in event_dispatch_epoll_worker.lto_priv () from /usr/local/lib/libglusterfs.so.0
#3 0x0000155554a7822a in ?? () from /usr/lib64/libc.so.6
#4 0x0000155554afacf0 in ?? () from /usr/lib64/libc.so.6
(gdb) bt
#0 0x0000155555473744 in iobuf_get2 () from /usr/local/lib/libglusterfs.so.0
#1 0x00001555532fb49c in socket_event_handler () from /usr/local/lib/glusterfs/11.1/rpc-transport/socket.so
#2 0x00001555554a836e in event_dispatch_epoll_worker.lto_priv () from /usr/local/lib/libglusterfs.so.0
#3 0x0000155554a7822a in ?? () from /usr/lib64/libc.so.6
#4 0x0000155554afacf0 in ?? () from /usr/lib64/libc.so.6
(gdb) f 0
#0 0x0000155555473744 in iobuf_get2 () from /usr/local/lib/libglusterfs.so.0
from glusterfs.
mohit84
commented on May 18, 2024
Please generate a coredump and attach a coredump to gdb and then run a command "thread apply all bt full" instead of running a cli process through gdb.
from glusterfs.
wonderful1984
commented on May 18, 2024
core-glfs_epoll001-318471-1711336118.zip
from glusterfs.
wonderful1984
commented on May 18, 2024
(gdb) set pagination 0
(gdb) thread apply all bt full
Thread 5 (Thread 0x14bdb8622640 (LWP 318474)):
#0 0x000014bdba044d92 in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
#1 0x000014bdba047934 in pthread_cond_timedwait () from /usr/lib64/libc.so.6
No symbol table info available.
#2 0x000000000040cca4 in cli_cmd_submit ()
No symbol table info available.
#3 0x000000000044fb8a in cli_to_glusterd.constprop ()
No symbol table info available.
#4 0x000000000042afff in gf_cli_status_volume_all.lto_priv ()
No symbol table info available.
#5 0x0000000000416c94 in cli_cmd_volume_status_cbk ()
No symbol table info available.
#6 0x000000000040c017 in cli_cmd_process ()
No symbol table info available.
#7 0x000000000040c1dc in cli_batch ()
No symbol table info available.
#8 0x000014bdba04822a in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
#9 0x000014bdba0cacf0 in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
Thread 4 (Thread 0x14bdb8421640 (LWP 318475)):
#0 0x000014bdba0ca2be in epoll_wait () from /usr/lib64/libc.so.6
No symbol table info available.
#1 0x000014bdbaa7823c in event_dispatch_epoll_worker.lto_priv () from /usr/local/lib/libglusterfs.so.0
No symbol table info available.
#2 0x000014bdba04822a in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
#3 0x000014bdba0cacf0 in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
Thread 3 (Thread 0x14bdb8823640 (LWP 318472)):
#0 0x000014bdba044d92 in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
#1 0x000014bdba047934 in pthread_cond_timedwait () from /usr/lib64/libc.so.6
No symbol table info available.
#2 0x000014bdbaa285a8 in gf_timer_proc () from /usr/local/lib/libglusterfs.so.0
No symbol table info available.
#3 0x000014bdba04822a in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
#4 0x000014bdba0cacf0 in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
Thread 2 (Thread 0x14bdb9b096c0 (LWP 318471)):
#0 0x000014bdba044d92 in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
#1 0x000014bdba049b83 in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
#2 0x000014bdbaa79f1f in event_dispatch_epoll.lto_priv () from /usr/local/lib/libglusterfs.so.0
No symbol table info available.
#3 0x0000000000409183 in main ()
No symbol table info available.
Thread 1 (Thread 0x14bdb8220640 (LWP 318476)):
#0 0x000014bdbaa43744 in iobuf_get2 () from /usr/local/lib/libglusterfs.so.0
No symbol table info available.
#1 0x000014bdb88cb49c in socket_event_handler () from /usr/local/lib/glusterfs/11.1/rpc-transport/socket.so
No symbol table info available.
#2 0x000014bdbaa7836e in event_dispatch_epoll_worker.lto_priv () from /usr/local/lib/libglusterfs.so.0
No symbol table info available.
#3 0x000014bdba04822a in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
#4 0x000014bdba0cacf0 in ?? () from /usr/lib64/libc.so.6
No symbol table info available.
from glusterfs.
mohit84
commented on May 18, 2024
I think you need to install source after enable debug symbol, that;s why gdb is not able to fetch the symbol at specific address.
from glusterfs.
wonderful1984
commented on May 18, 2024
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/gluster...
[New LWP 1996244]
[New LWP 1996195]
[New LWP 1996201]
[New LWP 1996243]
[New LWP 1996242]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
Core was generated by `gluster vol status all clients'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 iobuf_get_from_stdalloc (page_size=3247444, iobuf_pool=0x0) at iobuf.c:423
423 list_for_each_entry(trav, &iobuf_pool->arenas[IOBUF_ARENA_MAX_INDEX], list)
[Current thread is 1 (Thread 0x14e328f87640 (LWP 1996244))]
Missing separate debuginfos, use: dnf debuginfo-install e2fsprogs-1.46.4-17.oe2203sp1.x86_64 keyutils-libs-1.6.3-4.oe2203sp1.x86_64 krb5-libs-1.19.2-6.oe2203sp1.x86_64 libgcc-10.3.1-20.oe2203sp1.x86_64 libselinux-3.3-2.oe2203sp1.x86_64 libstdc++-10.3.1-20.oe2203sp1.x86_64 libtirpc-1.3.2-3.oe2203sp1.x86_64 libuuid-2.37.2-13.oe2203sp1.x86_64 libxml2-2.9.14-5.oe2203sp1.x86_64 ncurses-libs-6.3-5.oe2203sp1.x86_64 openssl-libs-1.1.1m-20.oe2203sp1.x86_64 pcre2-10.39-6.oe2203sp1.x86_64 readline-8.1-2.oe2203sp1.x86_64 userspace-rcu-0.13.0-5.oe2203sp1.x86_64 xz-libs-5.2.5-3.oe2203sp1.x86_64 zlib-1.2.11-22.oe2203sp1.x86_64
(gdb) set pagination 0
(gdb) thread apply all bt full
Thread 5 (Thread 0x14e329389640 (LWP 1996242)):
#0 0x000014e32adaad92 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x14e3293878b0, op=393, expected=0, futex_word=0x46224c <cond+44>) at futex-internal.c:57
sc_cancel_oldtype = 0
__arg6 = <optimized out>
__arg3 = <optimized out>
_a5 = <optimized out>
_a2 = <optimized out>
sc_ret = <optimized out>
__arg4 = <optimized out>
__arg1 = <optimized out>
_a6 = <optimized out>
_a3 = <optimized out>
resultvar = <optimized out>
__arg5 = <optimized out>
__arg2 = <optimized out>
_a4 = <optimized out>
_a1 = <optimized out>
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x46224c <cond+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x14e3293878b0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
err = <optimized out>
clockbit = 256
op = 393
#2 0x000014e32adaadff in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x46224c <cond+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x14e3293878b0, private=private@entry=0) at futex-internal.c:139
No locals.
#3 0x000014e32adad934 in __pthread_cond_wait_common (abstime=0x14e3293878b0, clockid=0, mutex=0x4621e0 <cond_mutex>, cond=0x462220 <cond>) at pthread_cond_wait.c:504
spin = 0
buffer = {__routine = 0x14e32adad3a0 <__condvar_cleanup_waiting>, __arg = 0x14e329387840, __canceltype = 4591584, __prev = 0x0}
cbuffer = {wseq = 3, cond = 0x462220 <cond>, mutex = 0x4621e0 <cond_mutex>, private = 0}
err = <optimized out>
g = 1
flags = <optimized out>
g1_start = <optimized out>
maxspin = 0
signals = <optimized out>
result = 0
wseq = 3
seq = 1
private = 0
maxspin = <optimized out>
err = <optimized out>
result = <optimized out>
wseq = <optimized out>
g = <optimized out>
seq = <optimized out>
flags = <optimized out>
private = <optimized out>
signals = <optimized out>
done = <optimized out>
g1_start = <optimized out>
spin = <optimized out>
buffer = <optimized out>
cbuffer = <optimized out>
s = <optimized out>
#4 ___pthread_cond_timedwait64 (cond=cond@entry=0x462220 <cond>, mutex=mutex@entry=0x4621e0 <cond_mutex>, abstime=abstime@entry=0x14e3293878b0) at pthread_cond_wait.c:644
flags = <optimized out>
clockid = 0
#5 0x000000000040aec2 in cli_cmd_await_response (time=time@entry=120) at cli-cmd.c:265
ts = {tv_sec = 1711599554, tv_nsec = 858221356}
ret = 0
#6 0x000000000040b462 in cli_cmd_submit (rpc=rpc@entry=0x0, req=req@entry=0x14e3293879a0, frame=frame@entry=0x17b2568, prog=prog@entry=0x460fe0 <cli_prog>, procnum=procnum@entry=27, this=this@entry=0x14e32b85e560 <global_xlator>, cbkfn=0x429a20 <gf_cli_status_cbk>, xdrproc=0x4084c0 <xdr_gf_cli_req@plt>) at cli-cmd.c:372
ret = 0
timeout = 120
state = <optimized out>
__FUNCTION__ = "cli_cmd_submit"
#7 0x000000000041c942 in cli_to_glusterd (req=req@entry=0x14e3293879a0, frame=frame@entry=0x17b2568, cbkfn=cbkfn@entry=0x429a20 <gf_cli_status_cbk>, dict=dict@entry=0x17a6428, procnum=procnum@entry=27, this=this@entry=0x14e32b85e560 <global_xlator>, prog=0x460fe0 <cli_prog>, xdrproc=<optimized out>) at cli-rpc-ops.c:10668
ret = <optimized out>
len = <optimized out>
cmd = <optimized out>
i = <optimized out>
words = <optimized out>
local = <optimized out>
state = 0x7ffec163ba50
__FUNCTION__ = "cli_to_glusterd"
#8 0x000000000041d633 in gf_cli_status_volume (frame=frame@entry=0x17b2568, this=this@entry=0x14e32b85e560 <global_xlator>, data=data@entry=0x17a6428) at cli-rpc-ops.c:7637
req = {dict = {dict_len = 86, dict_val = 0x17a4388 ""}}
ret = -1
dict = 0x17a6428
__FUNCTION__ = "gf_cli_status_volume"
#9 0x000000000041d868 in gf_cli_status_volume_all (frame=frame@entry=0x17b2568, this=0x14e32b85e560 <global_xlator>, data=data@entry=0x17a6328) at cli-rpc-ops.c:7728
i = 0
ret = <optimized out>
vol_count = 1
cmd = 258
key = "vol0", '\000' <repeats 1019 times>
volname = 0x17cc6a8 "backup-img"
vol_dict = 0x17a64a8
dict = 0x17a6428
local = 0x17b6708
state = 0x7ffec163ba50
__PRETTY_FUNCTION__ = "gf_cli_status_volume_all"
__FUNCTION__ = "gf_cli_status_volume_all"
xml_end = <optimized out>
#10 0x0000000000410e73 in cli_cmd_volume_status_cbk (state=<optimized out>, word=<optimized out>, words=<optimized out>, wordcount=<optimized out>) at cli-cmd-volume.c:2315
ret = <optimized out>
proc = 0x4611e0 <gluster_cli_actors+448>
frame = 0x17b2568
dict = 0x17a6328
cmd = 514
local = 0x17b6708
out = <optimized out>
__FUNCTION__ = "cli_cmd_volume_status_cbk"
#11 0x000000000040b09e in cli_cmd_process (state=0x7ffec163ba50, argc=4, argv=0x7ffec163bcb0) at cli-cmd.c:131
ret = <optimized out>
word = 0x1778af0
next = <optimized out>
i = <optimized out>
__FUNCTION__ = "cli_cmd_process"
#12 0x000000000040ab5c in cli_batch (d=<optimized out>) at input.c:29
state = <optimized out>
ret = 0
__FUNCTION__ = "cli_batch"
#13 0x000014e32adae22a in start_thread (arg=<optimized out>) at pthread_create.c:443
ret = <optimized out>
pd = <optimized out>
out = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140732142958640, 7385284747875049934, 22965881706048, 0, 22965909118816, 140732142958944, 5745360750633017806, 5745353891887440334}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#14 0x000014e32ae30cf0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.
Thread 4 (Thread 0x14e329188640 (LWP 1996243)):
#0 0x000014e32ae302be in epoll_wait (epfd=3, events=events@entry=0x14e329186e84, maxevents=maxevents@entry=1, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
sc_ret = -4
sc_cancel_oldtype = 0
sc_ret = <optimized out>
#1 0x000014e32b7d0e24 in event_dispatch_epoll_worker (data=0x17cc4a8) at event-epoll.c:715
event = {events = 4, data = {ptr = 0x100000000, fd = 0, u32 = 0, u64 = 4294967296}}
ret = <optimized out>
ev_data = 0x17cc4a8
event_pool = 0x1782028
myindex = 1
timetodie = 0
gen = 0
poller_death_notify = {next = 0x0, prev = 0x0}
slot = 0x0
tmp = 0x0
__FUNCTION__ = "event_dispatch_epoll_worker"
out = <optimized out>
#2 0x000014e32adae22a in start_thread (arg=<optimized out>) at pthread_create.c:443
ret = <optimized out>
pd = <optimized out>
out = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140732142957984, 7385284747875049934, 22965879604800, 0, 22965909118816, 140732142958288, 5745361026047795662, 5745353891887440334}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#3 0x000014e32ae30cf0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.
Thread 3 (Thread 0x14e32958a640 (LWP 1996201)):
#0 0x000014e32adaad92 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x14e329588ea0, op=137, expected=0, futex_word=0x17b629c) at futex-internal.c:57
sc_cancel_oldtype = 0
__arg6 = <optimized out>
__arg3 = <optimized out>
_a5 = <optimized out>
_a2 = <optimized out>
sc_ret = <optimized out>
__arg4 = <optimized out>
__arg1 = <optimized out>
_a6 = <optimized out>
_a3 = <optimized out>
resultvar = <optimized out>
__arg5 = <optimized out>
__arg2 = <optimized out>
_a4 = <optimized out>
_a1 = <optimized out>
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x17b629c, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x14e329588ea0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87
err = <optimized out>
clockbit = 0
op = 137
#2 0x000014e32adaadff in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x17b629c, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x14e329588ea0, private=private@entry=0) at futex-internal.c:139
No locals.
#3 0x000014e32adad934 in __pthread_cond_wait_common (abstime=0x14e329588ea0, clockid=1, mutex=0x17b6248, cond=0x17b6270) at pthread_cond_wait.c:504
spin = 0
buffer = {__routine = 0x14e32adad3a0 <__condvar_cleanup_waiting>, __arg = 0x14e329588e30, __canceltype = 0, __prev = 0x0}
cbuffer = {wseq = 7, cond = 0x17b6270, mutex = 0x17b6248, private = 0}
err = <optimized out>
g = 1
flags = <optimized out>
g1_start = <optimized out>
maxspin = 0
signals = <optimized out>
result = 0
wseq = 7
seq = 3
private = 0
maxspin = <optimized out>
err = <optimized out>
result = <optimized out>
wseq = <optimized out>
g = <optimized out>
seq = <optimized out>
flags = <optimized out>
private = <optimized out>
signals = <optimized out>
done = <optimized out>
g1_start = <optimized out>
spin = <optimized out>
buffer = <optimized out>
cbuffer = <optimized out>
s = <optimized out>
#4 ___pthread_cond_timedwait64 (cond=cond@entry=0x17b6270, mutex=mutex@entry=0x17b6248, abstime=abstime@entry=0x14e329588ea0) at pthread_cond_wait.c:644
flags = <optimized out>
clockid = 1
#5 0x000014e32b788a38 in gf_timer_proc (data=0x17b6238) at timer.c:140
now = {tv_sec = 1127334, tv_nsec = 67471480}
reg = 0x17b6238
event = 0x17ac178
tmp = 0x0
old_THIS = <optimized out>
#6 0x000014e32adae22a in start_thread (arg=<optimized out>) at pthread_create.c:443
ret = <optimized out>
pd = <optimized out>
out = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140732142957904, 7385284747875049934, 22965883807296, 17, 22965909118816, 140732142958208, 5745361570434900430, 5745353891887440334}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#7 0x000014e32ae30cf0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.
Thread 2 (Thread 0x14e32a86f6c0 (LWP 1996195)):
#0 0x000014e32adaad92 in __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=1996243, futex_word=0x14e329188910) at futex-internal.c:57
sc_cancel_oldtype = 0
__arg6 = <optimized out>
__arg3 = <optimized out>
_a5 = <optimized out>
_a2 = <optimized out>
sc_ret = <optimized out>
__arg4 = <optimized out>
__arg1 = <optimized out>
_a6 = <optimized out>
_a3 = <optimized out>
resultvar = <optimized out>
__arg5 = <optimized out>
__arg2 = <optimized out>
_a4 = <optimized out>
_a1 = <optimized out>
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x14e329188910, expected=1996243, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at futex-internal.c:87
err = <optimized out>
clockbit = 256
op = 265
#2 0x000014e32adaadff in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x14e329188910, expected=<optimized out>, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at futex-internal.c:139
No locals.
#3 0x000014e32adafb83 in __pthread_clockjoin_ex (threadid=22965879604800, thread_return=thread_return@entry=0x0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, block=block@entry=true) at pthread_join_common.c:102
ret = <optimized out>
_buffer = {__routine = 0x14e32adafa60 <cleanup>, __arg = 0x14e329188c68, __canceltype = 0, __prev = 0x0}
tid = <optimized out>
pd = 0x14e329188640
self = <optimized out>
result = 0
pd_result = <optimized out>
#4 0x000014e32adafa53 in ___pthread_join (threadid=<optimized out>, thread_return=thread_return@entry=0x0) at pthread_join.c:25
No locals.
#5 0x000014e32b7d06df in event_dispatch_epoll (event_pool=0x1782028) at event-epoll.c:809
i = <optimized out>
t_id = 22965877503552
pollercount = 2
ret = 0
ev_data = <optimized out>
__FUNCTION__ = "event_dispatch_epoll"
#6 0x0000000000409187 in main (argc=<optimized out>, argv=<optimized out>) at cli.c:811
state = {argc = 4, argv = 0x7ffec163bcb0, debug = 0 '\000', ctx = 0x1772000, rpc = 0x17ca028, quotad_rpc = 0x0, tree = {state = 0x7ffec163ba50, root = {tree = 0x7ffec163ba80, word = 0x0, filler = 0x0, match = 0x0, cbkfn = 0x0, desc = 0x0, pattern = 0x0, nextwords_cnt = 11, nextwords = 0x17ac310}}, input = 22965881706048, prompt = 0x0, rl_enabled = 0, rl_async = 0, rl_processing = 0, matches = 0x0, matchesp = 0x0, remote_host = 0x0, remote_port = 0, mode = 0, await_connected = 120, default_conn_timeout = 120, log_file = 0x0, log_level = GF_LOG_NONE, glusterd_sock = 0x0, address_family = 0x0}
ret = 0
ctx = 0x1772000
out = <optimized out>
__FUNCTION__ = "main"
Thread 1 (Thread 0x14e328f87640 (LWP 1996244)):
#0 iobuf_get_from_stdalloc (page_size=3247444, iobuf_pool=0x0) at iobuf.c:423
iobuf = 0x0
iobuf_arena = 0x0
trav = 0x0
#1 iobuf_get2 (iobuf_pool=0x0, page_size=3247444) at iobuf.c:493
iobuf = 0x0
rounded_size = <optimized out>
index = <optimized out>
__FUNCTION__ = "iobuf_get2"
post_unlock = <optimized out>
#2 0x000014e32962fcf1 in socket_proto_state_machine (pollin=<synthetic pointer>, this=0x17ae5a8) at socket.c:2234
ret = <optimized out>
iobuf = <optimized out>
frag = <optimized out>
priv = <optimized out>
iobref = 0x0
vector = {{iov_base = 0x17ca560, iov_len = 256}, {iov_base = 0x0, iov_len = 0}}
in = <optimized out>
ret = <optimized out>
priv = <optimized out>
iobuf = <optimized out>
iobref = <optimized out>
vector = <optimized out>
in = <optimized out>
frag = <optimized out>
__FUNCTION__ = "socket_proto_state_machine"
out = <optimized out>
count = <optimized out>
#3 socket_event_poll_in (notify_handled=true, this=0x17ae5a8) at socket.c:2384
ret = -1
pollin = 0x0
priv = 0x1812028
ctx = 0x1772000
ret = <optimized out>
pollin = <optimized out>
priv = <optimized out>
ctx = <optimized out>
#4 socket_event_handler (event_thread_died=0, poll_err=0, poll_out=<optimized out>, poll_in=<optimized out>, data=0x17ae5a8, gen=1, idx=0, fd=<optimized out>) at socket.c:2790
this = <optimized out>
ret = <optimized out>
ctx = <optimized out>
notify_handled = <optimized out>
priv = 0x1812028
socket_closed = <optimized out>
this = <optimized out>
priv = <optimized out>
ret = <optimized out>
ctx = <optimized out>
socket_closed = <optimized out>
notify_handled = <optimized out>
__FUNCTION__ = "socket_event_handler"
out = <optimized out>
sock_type = <optimized out>
sa = <optimized out>
#5 socket_event_handler (fd=fd@entry=6, idx=idx@entry=0, gen=gen@entry=1, data=data@entry=0x17ae5a8, poll_in=<optimized out>, poll_out=<optimized out>, poll_err=0, event_thread_died=0) at socket.c:2710
this = 0x17ae5a8
priv = 0x0
ret = -1
ctx = 0x0
socket_closed = false
notify_handled = false
__FUNCTION__ = "socket_event_handler"
out = <optimized out>
sock_type = <optimized out>
sa = <optimized out>
#6 0x000014e32b7d0f4d in event_dispatch_epoll_handler (event=0x14e328f85e84, event_pool=0x1782028) at event-epoll.c:614
handler = <optimized out>
gen = <optimized out>
pre_unlock = <optimized out>
slot = 0x17860e8
data = <optimized out>
ret = 0
fd = 6
ev_data = 0x14e328f85e88
idx = 0
handled_error_previously = false
ev_data = <optimized out>
slot = <optimized out>
handler = <optimized out>
data = <optimized out>
idx = <optimized out>
gen = <optimized out>
ret = <optimized out>
fd = <optimized out>
handled_error_previously = <optimized out>
__FUNCTION__ = "event_dispatch_epoll_handler"
pre_unlock = <optimized out>
out = <optimized out>
#7 event_dispatch_epoll_worker (data=0x17cc468) at event-epoll.c:725
event = {events = 1, data = {ptr = 0x100000000, fd = 0, u32 = 0, u64 = 4294967296}}
ret = <optimized out>
ev_data = 0x17cc468
event_pool = 0x1782028
myindex = 2
timetodie = 0
gen = 0
poller_death_notify = {next = 0x0, prev = 0x0}
slot = 0x0
tmp = 0x0
__FUNCTION__ = "event_dispatch_epoll_worker"
out = <optimized out>
#8 0x000014e32adae22a in start_thread (arg=<optimized out>) at pthread_create.c:443
ret = <optimized out>
pd = <optimized out>
out = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140732142957984, 7385284747875049934, 22965877503552, 0, 22965909118816, 140732142958288, 5745358002927690190, 5745353891887440334}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#9 0x000014e32ae30cf0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
No locals.
(gdb)
from glusterfs.
mohit84
commented on May 18, 2024
As per backtrace it is showing cli process is getting crashed while trying to access a iobuf_pool list of entries, as we can see
the function ( iobuf_get_from_stdalloc (page_size=3247444, iobuf_pool=0x0)) is calling with NULL iobuf_pool. It is a basic code path and no one would be able to run any cli command if iobuf_pool is NULL. The iobuf_pool is allocated by glusterfs_ctx_defaults_init the function call by main function as a part of cli.c. It is the starting function that allocates pool structure. Are you getting any error in cli.logs while running a command?
from glusterfs.
wonderful1984
commented on May 18, 2024
glusterfs.log
[2024-03-28 05:02:16.592514 +0000] I [MSGID: 106061] [glusterd-utils.c:9576:glusterd_volume_status_copy_to_op_ctx_dict] 0-management: Dict get failed [{Key=count}]
[2024-03-28 05:02:16.592667 +0000] I [MSGID: 106499] [glusterd-handler.c:4535:__glusterd_handle_status_volume] 0-management: Received status volume req for volume backup-img
cli.log
[2024-03-28 05:02:16.515610 +0000] I [cli.c:788:main] 0-cli: Started running /usr/local/sbin/gluster with version 11.1
[2024-03-28 05:02:16.591492 +0000] I [MSGID: 101188] [event-epoll.c:643:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}]
[2024-03-28 05:02:16.591771 +0000] I [MSGID: 101188] [event-epoll.c:643:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}]
from glusterfs.
Related Issues (20)
- Strange issue with permission denied and bizarre mtime HOT 5
- sq: rename may cause statistical issue HOT 1
- sq: df show wrong usage after file deleted
- Gluster Linked File Issues during Healing HOT 1
- qemu-img create via libfapi is slow HOT 8
- [BUG] Geo-rep transfers large files multiple times while using Tar+ssh HOT 3
- Errors creating files via qemu-img / libgfapi
- The rights to folders and files from the glusterfs client are flying off HOT 3
- glusterfs: symbol lookup error: /lib/x86_64-linux-gnu/libgfrpc.so.0: undefined symbol: gf_async_ctrl HOT 1
- Transport endpoint is not connected HOT 5
- Persistent File Staleness Errors in GlusterFS Volume HOT 2
- dentry_hashsize HOT 27
- inode->in_lru_list is not really used
- Do not update inode stats in real time - collect during dump HOT 4
- ZFS xattr conflict due to forbidden 'trusted' prefix HOT 3
- Symlinking to a file with a long filepath results in a broken link HOT 4
- logging.warn is removed in Python 3.13
- Error in count inode glusterfs HOT 1
- Glustrefs geo replication anytime consistency question
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glusterfs.