cgdb freezes at startup (on ARM) about cgdb HOT 25 CLOSED

On Fri, May 30, 2014 at 05:24:06AM -0700, kraiskil wrote:

cgdb freezes on startup after reading symbols:

cgdb terminal output:
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /home/kraiskil/hello...done.

Backtrace:
(gdb) bt
#0 0xb6e5e4fc in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x0000af5a in main_loop () at cgdb.c:1338
#2 main (argc=, argv=) at cgdb.c:1737

Only a "kill -9" makes it stop. This seems an ARM issue, I use cgdb extensively (my favourite UI to gdb, thanks!), and have seen it on at least two ARM boxes, never on a x86 box. Could be it is a race that is triggered because ARM is generally slower than x86, don't know.
The testprogram cgdb is debugging is a "int main(void){return 0;}, compiled with 'gcc -O0 -g hello.c -o hello'. The hello-program debugs fine on plain gdb on the same system.

What is the contents of,

~/.tgdb/a2_tgdb_debug.txt

You are sure the exact same scenario works fine with vanilla GDB?
(The same version of GDB that cgdb is using?)

Thanks,
Bob Rossi

from cgdb.

Yes, working plain gdb is version 7.6.1-ubuntu, same that cgdb picks up.

Contents of a2_tgdb_debug.txt:
$ cat .tgdb/a2_tgdb_debug.txt
GNU gdb (GDB) 7.6.1-ubuntu(\r)(\n)
Copyright (C) 2013 Free Software Foundation, Inc.(\r)(\n)
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html(\r)(\n)
This is free software: you are free to change and redistribute it.(\r)(\n)
There is NO WARRANTY, to the extent permitted by law. Type "show copying"(\r)(\n)
and "show warranty" for details.(\r)(\n)
This GDB was configured as "arm-linux-gnueabihf".(\r)(\n)
For bug reporting instructions, please see:(\r)(\n)
http://www.gnu.org/software/gdb/bugs/...(\r)(\n)
Reading symbols from /home/kraiskil/hello...done.(\r)(\n)

from cgdb.

On Fri, May 30, 2014 at 08:36:28AM -0700, kraiskil wrote:

Yes, working plain gdb is version 7.6.1-ubuntu, same that cgdb picks up.

Contents of a2_tgdb_debug.txt:
$ cat .tgdb/a2_tgdb_debug.txt
GNU gdb (GDB) 7.6.1-ubuntu(\r)(\n)
Copyright (C) 2013 Free Software Foundation, Inc.(\r)(\n)
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html(\r)(\n)
This is free software: you are free to change and redistribute it.(\r)(\n)
There is NO WARRANTY, to the extent permitted by law. Type "show copying"(\r)(\n)
and "show warranty" for details.(\r)(\n)
This GDB was configured as "arm-linux-gnueabihf".(\r)(\n)
For bug reporting instructions, please see:(\r)(\n)
http://www.gnu.org/software/gdb/bugs/...(\r)(\n)
Reading symbols from /home/kraiskil/hello...done.(\r)(\n)

OK. Looks like annotations is broken with this version of GDB.

Try running the below command and send me the output.

(I've showed you the expected output)

Thanks,
Bob Rossi

$ gdb --annotate=2 ./main
GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./main...done.

pre-prompt
(gdb)
prompt

from cgdb.

Seems to be the same. However both the "pre-prompt" and "prompt" are prefixed with two UTF substitute-characters (0x001A) - copying the output to github lost them.

kraiskil@notdroid:~$ gdb --annotate=2 hello
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /home/kraiskil/hello...done.

��pre-prompt
(gdb)
��prompt

from cgdb.

Just FYI,

On Raspbian GNU/Linux 7 (Debian 7.4) with these versions installed:

cgdb 0.6.6-2
gdb 7.4.1+dfsg-0.1

and gdb reporting '"This GDB was configured as "arm-linux-gnueabihf"'.

I have no problems and cgdb works fine on the hello program (and
others).

from cgdb.

Tried with compiling gdb 7.7.1 from source. Using this new gdb, cgdb (tried both Ubuntu-packaged v0.6.7 and compiled from git yesterday) does start & run properly, but every now-and-then it freezes when quitting.
When it happens, I'm getting a similar backtrace from cgdb as in issue #46. Attaching to the slave gdb in this case says the gdb is already a zombie.
So this freeze-upon-startup might be an Ubuntu or gdb issue.

from cgdb.

Duplicate of #46

from cgdb.

On Sun, Jun 01, 2014 at 03:18:42AM -0700, kraiskil wrote:

Tried with compiling gdb 7.7.1 from source. Using this new gdb, cgdb (tried both Ubuntu-packaged v0.6.7 and compiled from git yesterday) does start & run properly, but every now-and-then it freezes when quitting.

Finally fixed upstream.

from cgdb.

Just for the record: I updated my system to Ubuntu 14.04 which ships gdb "Ubuntu 7.7-0ubuntu3.1"
The freeze-upon-start issue happens when using this version of gdb, with both Ubuntu-shipped cgdb and the compiled-from-source one (version cbfbb5f).
Using cgdb with gdb 7.7.1 compiled from source works fine.

Oh, and I noticed this has actually nothing to do with reading symbols. Just starting cgdb without any command line options causes a freeze.

from cgdb.

On Tue, Aug 05, 2014 at 02:34:15AM -0700, kraiskil wrote:

Just for the record: I updated my system to Ubuntu 14.04 which ships gdb "Ubuntu 7.7-0ubuntu3.1"
The freeze-upon-start issue happens when using this version of gdb, with both Ubuntu-shipped cgdb and the compiled-from-source one (version cbfbb5f).
Using cgdb with gdb 7.7.1 compiled from source works fine.

Oh, and I noticed this has actually nothing to do with reading symbols. Just starting cgdb without any command line options causes a freeze.

Ouch. I would expect this with the latest release of cgdb but not with
cgdb from source (version cbfbb5f).

I'm suprised with all of these issues we are finding.

Can you provide me with the backtrace of cgdb when it is hung?
You'll have to rebuild it with debug support.

Also, the contents of ~/.tgdb/a2_tgdb_debug.txt would be helpful.

Thanks,
Bob Rossi

from cgdb.

The backtrace and the debug.txt were the same as reported earlier in this issue.
I tried to compile the gdb from the sources of gdb that Ubuntu ships - no freeze.
I.e. the issue here is due to something in how Ubuntu packages gdb - patch or configure option.
I reported this as an Ubuntu-bug (as its out-of-the-packet cgdb does not work): https://bugs.launchpad.net/ubuntu/+source/cgdb/+bug/1357378 Perhaps they can give some light to this issue.

from cgdb.

Seems I am not alone with this issue: https://bugs.launchpad.net/ubuntu/+source/cgdb/+bug/1030088

from cgdb.

On Fri, Aug 15, 2014 at 10:54:39AM -0700, kraiskil wrote:

Seems I am not alone with this issue: https://bugs.launchpad.net/ubuntu/+source/cgdb/+bug/1030088

Can you run this command,
echo 'q' | gdb --annotate=2 | hexdump -c
and send in the output?

Thanks,
Bob Rossi

from cgdb.

Here is the output of both breaking and working gdb: https://gist.github.com/kraiskil/d82c4f7b1c78b3deaeaa

from cgdb.

Unfortunately, this was not helpful. My next step is to do an analysis of the source code to see if I can find anything. If that isn't helping, I'm going to have to provide you with a custom build with debugging information in it.

from cgdb.

Ok, thanks.

I was looking into the buildlogs of gdb: Ubuntu gdb package build-log (this causes the error)
https://launchpadlibrarian.net/175062748/buildlog_ubuntu-trusty-armhf.gdb_7.7-0ubuntu3.1_UPLOADING.txt.gz
On Fedora (20) ARM the gdb works:
http://kojipkgs.fedoraproject.org/packages/gdb/7.7.1/18.fc20/data/logs/armv7hl/build.log
And of course a plain ./configure --prefix=/opt/gdb && make && make install creates a working build.
But I cannot spot an obvious difference that would cause the fail in the ./configure lines.

from cgdb.

On Wed, Aug 20, 2014 at 10:07:53PM -0700, kraiskil wrote:

Ok, thanks.

I was looking into the buildlogs of gdb: Ubuntu gdb package build-log (this causes the error)
https://launchpadlibrarian.net/175062748/buildlog_ubuntu-trusty-armhf.gdb_7.7-0ubuntu3.1_UPLOADING.txt.gz
On Fedora (20) ARM the gdb works:
http://kojipkgs.fedoraproject.org/packages/gdb/7.7.1/18.fc20/data/logs/armv7hl/build.log
And of course a plain ./configure --prefix=/opt/gdb && make && make install creates a working build.
But I cannot spot an obvious difference that would cause the fail in the ./configure lines.

Thanks for doing this. I'll review the logs.

Do you know if there is a link to a version of ubuntu that I could
install on a virtual machine to attempt to reproduce the problem?

Thanks,
Bob Rossi

from cgdb.

I don't know of a straight-forward way of running Ubuntu on a virtual ARM. Qemu would be the way to go there, but there seem to be quite a few steps. Also, I am still not sure that this would not be a race: one of the comments in the Ubuntu bug I linked above is that this happens everytime the debuggee resides on a NFS share - so could be that the qemu-arm wouldn't have this issue. (I have access to an Fedora 20-ARM that does not exhibit this problem, after all...).

That said - I tracked down the offending configuration option passed to gdb: --with-system-readline. Adding this option to gdb ./configure builds a gdb with which cgdb freezes. And similarly - taking all the ubuntu build settings in the link above, but removing this one option builds a "working" gdb. So - a readline bug, or just something cgdb should take into account - not sure now.

Also - seems both Fedora 20 and Ubuntu 14.04 have the same version of readlne - 6.3. The Fedora-ARM box I have has a clock speed of 1/3 compared to my Ubuntu ARM box, which again suggests this might be a generic race issue, not something ARM specific.

from cgdb.

On Tue, Aug 26, 2014 at 11:18:06AM -0700, kraiskil wrote:

I don't know of a straight-forward way of running Ubuntu on a virtual ARM. Qemu would be the way to go there, but there seem to be quite a few steps. Also, I am still not sure that this would not be a race: one of the comments in the Ubuntu bug I linked above is that this happens everytime the debuggee resides on a NFS share - so could be that the qemu-arm wouldn't have this issue. (I have access to an Fedora 20-ARM that does not exhibit this problem, after all...).

OK. The NFS comment was a misunderstanding. CGDB works fine on NFS.

That said - I tracked down the offending configuration option passed to gdb: --with-system-readline. Adding this option to gdb ./configure builds a gdb with which cgdb freezes. And similarly - taking all the ubuntu build settings in the link above, but removing this one option builds a "working" gdb. So - a readline bug, or just something cgdb should take into account - not sure now.

I'm trying to reproduce this now.

Also - seems both Fedora 20 and Ubuntu 14.04 have the same version of readlne - 6.3. The Fedora-ARM box I have has a clock speed of 1/3 compared to my Ubuntu ARM box, which again suggests this might be a generic race issue, not something ARM specific.

Good point.

Bob Rossi

from cgdb.

As an additional data point, Raspbian (Debian for Raspberry Pi) has
readline 6.2 and I've yet to have cgdb freeze up on it.

On Aug 26, 2014, at 15:29, brasko wrote:

On Tue, Aug 26, 2014 at 11:18:06AM -0700, kraiskil wrote:

I don't know of a straight-forward way of running Ubuntu on a
virtual ARM. Qemu would be the way to go there, but there seem to be
quite a few steps. Also, I am still not sure that this would not be
a race: one of the comments in the Ubuntu bug I linked above is that
this happens everytime the debuggee resides on a NFS share - so
could be that the qemu-arm wouldn't have this issue. (I have access
to an Fedora 20-ARM that does not exhibit this problem, after all...).

OK. The NFS comment was a misunderstanding. CGDB works fine on NFS.

That said - I tracked down the offending configuration option
passed to gdb: --with-system-readline. Adding this option to
gdb ./configure builds a gdb with which cgdb freezes. And similarly

• taking all the ubuntu build settings in the link above, but
  removing this one option builds a "working" gdb. So - a readline
  bug, or just something cgdb should take into account - not sure now.

If you build readline 6.2 on the system that hangs and then use
LD_LIBRARY_PATH to make readline 6.2 load instead of 6.3, does the
stock cgdb+gdb stop freezing?

I'm trying to reproduce this now.

Also - seems both Fedora 20 and Ubuntu 14.04 have the same version
of readlne - 6.3. The Fedora-ARM box I have has a clock speed of 1/3
compared to my Ubuntu ARM box, which again suggests this might be a
generic race issue, not something ARM specific.

Good point.

Bob Rossi

from cgdb.

On Tue, Aug 26, 2014 at 09:32:40PM -0700, mackyle wrote:

As an additional data point, Raspbian (Debian for Raspberry Pi) has
readline 6.2 and I've yet to have cgdb freeze up on it.

On Aug 26, 2014, at 15:29, brasko wrote:

On Tue, Aug 26, 2014 at 11:18:06AM -0700, kraiskil wrote:

I don't know of a straight-forward way of running Ubuntu on a
virtual ARM. Qemu would be the way to go there, but there seem to be
quite a few steps. Also, I am still not sure that this would not be
a race: one of the comments in the Ubuntu bug I linked above is that
this happens everytime the debuggee resides on a NFS share - so
could be that the qemu-arm wouldn't have this issue. (I have access
to an Fedora 20-ARM that does not exhibit this problem, after all...).

OK. The NFS comment was a misunderstanding. CGDB works fine on NFS.

That said - I tracked down the offending configuration option
passed to gdb: --with-system-readline. Adding this option to
gdb ./configure builds a gdb with which cgdb freezes. And similarly

• taking all the ubuntu build settings in the link above, but
  removing this one option builds a "working" gdb. So - a readline
  bug, or just something cgdb should take into account - not sure now.

If you build readline 6.2 on the system that hangs and then use
LD_LIBRARY_PATH to make readline 6.2 load instead of 6.3, does the
stock cgdb+gdb stop freezing?

Is there any chance you are using readline in vi mode?

Bob Rossi

from cgdb.

On Aug 27, 2014, at 06:30, brasko wrote:

On Tue, Aug 26, 2014 at 09:32:40PM -0700, mackyle wrote:

As an additional data point, Raspbian (Debian for Raspberry Pi) has
readline 6.2 and I've yet to have cgdb freeze up on it.

Is there any chance you are using readline in vi mode?

I assume that's directed to @kraiskil... But just in case it's not,
nope, I have not set editing-mode so it's in the default emacs mode.

from cgdb.

No, I have not set readline to vi mode.
A slight correction to the above, seems Fedora 20 has readline 6.2 after all.

To recap - platforms I've seen that exhibit the issue:
-Ubuntu 12.04 - readline 6.2-8, 1.2GHz Cortex-A9 (pandaboard)
-Ubuntu 13.10 - readline 6.2-9ubuntu1, 1.7GHz Cortex-A9 (odroid U3)
-Ubuntu 14.04 readline 6.3-4ubuntu2 (odroid U3)

Platforms that don't have the issue:
-Fedora 20 - readline 6.2, 667MHz Cortex-A9(zynq)
-Raspbian - readline 6.2, 850MHz ARM11 (R-Pi)

There aren't too many data or triangulation points, but from these the common factor is Ubuntu and/or higher perf ARM cores. (The Cortex-A9 has about 2x the MIPS/MHz compared to ARM11
I tried to install readline debug symbols to the Ubuntu 14.04 and attach another gdb to the cgdb child gdb, when the pair had hung up. This caused complaints about a corrupted stack. I'll try to build a few versions of readline from sources and see what happens.

from cgdb.

Tried with building readline from sources on the Ubuntu 14.04. With that, the Ubuntu-packaged cgdb&gdb worked.
I tried readline plain 6.2 and 6.3, and 6.3 with ubuntu patches and ./configure options. None of these caused a hang. I probably missed something when trying to double up the Ubuntu build of its readline...

from cgdb.

I was never able to reproduce this. I'm going to close it again unless I get further feedback.

from cgdb.