Problems compiling xdelta3 under solaris 10 using gcc 3.4.3 about xdelta HOT 10 CLOSED

I have had some progress with this issue. It works! 

Running xdelta3 with the -N option allows the program to run without crashing. 
I have
used this in production a few thousand times now and it has not failed. However 
I
have gone live with code which uses xdelta twice on the server side, it creates 
the
delta, applies the delta to create a temp file which is then 'diffed' against 
the
original newer file. This is quite an overhead, but I feel I cannot fully trust 
the
current version of xdelta3.

With a colleague of mine, PTadman, we attempted to locate what was going on in 
the
code. But found it too difficult to read!

Using the -v option also causes a crash, but in a different area of the program.

The -v option may be causing crashes due to printf thinking some integer type 
is a
different size than it really is.

I wish you could provide more info on the crashes you see.  I am not receiving 
bug
reports for x86 platforms these days.  3.0r fixes a bug with "-S djw", but 
otherwise
I've been working on performance improvements. I feel that you're having
Solaris-specific issues, and I would like them fixed. I should get a 64bit 
platform
of some sort to test on..

I have thought about adding an extra safety mode for encoding, where it would
"virtually" decode the result as the data is encoded, to verify during encoding 
that
the delta is correct w/o the extra step you mention.

Ok, hope this helps. Let me know if there is any more I can do; but I cant send 
core 
files. The test case was a copy of the GPL with and without a s/the/hte/g. So 
repeating a working diff again without the -N option we get a core dump.

ultra45(fergus)% xdelta3 -e -s gpl-not.html gpl.html diff.out
 Bus Error (core dumped)
ultra45(fergus)% gdb /usr/local/bin/xdelta3 core
 GNU gdb 6.6
 Copyright (C) 2006 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "sparc-sun-solaris2.10"...
 Reading symbols from /lib/libm.so.2...done.
 Loaded symbols for /lib/libm.so.2
 Reading symbols from /lib/libc.so.1...done.
 Loaded symbols for /lib/libc.so.1
 Reading symbols from /platform/SUNW,A70/lib/libc_psr.so.1...done.
 Loaded symbols for /platform/SUNW,A70/lib/libc_psr.so.1
 Reading symbols from /lib/ld.so.1...done.
 Loaded symbols for /lib/ld.so.1
 Core was generated by `xdelta3 -e -s gpl-not.html gpl.html diff.out'.
 Program terminated with signal 10, Bus error.
 #0  0x00024660 in xd3_scksum (
     seg=0x5a232 "These Terms to Your New Programs\n    
</a></span></dt></dl></div><div class=\"sect1\" lang=\"en\"><div 
class=\"titlepage\"><div><div><h2 class=\"title\" style=\"clear: both\"><a 
name=\"gpl-1\"></a>Preamble</h2></d"..., ln=4)
     at xdelta3.c:1637
 1637    xdelta3.c: No such file or directory.
         in xdelta3.c
 (gdb) where
 #0  0x00024660 in xd3_scksum (
     seg=0x5a232 "These Terms to Your New Programs\n    
</a></span></dt></dl></div><div class=\"sect1\" lang=\"en\"><div 
class=\"titlepage\"><div><div><h2 class=\"title\" style=\"clear: both\"><a 
name=\"gpl-1\"></a>Preamble</h2></d"..., ln=4)
     at xdelta3.c:1637
 #1  0x000247e4 in xd3_string_match_fastest (stream=0xffbfec50) at xdelta3.c:4876
 #2  0x000214ac in xd3_encode_input (stream=0xffbfec50) at xdelta3.c:3720
 #3  0x0002e4fc in main_input (cmd=CMD_ENCODE, ifile=0xffbff0b8, ofile=0xffbff080, 
sfile=0xffbff048) at xdelta3-main.h:2571
 #4  0x00030254 in main (argc=2, argv=0xffbff17c) at xdelta3-main.h:3221
 (gdb) quit
ultra45(fergus)%


Than using the -v option.

ultra45(fergus)% xdelta3 -v -e -s gpl-not.html gpl.html diff.out
 Segmentation Fault (core dumped)
ultra45(fergus)%
ultra45(fergus)% gdb /usr/local/bin/xdelta3 core
 GNU gdb 6.6
 Copyright (C) 2006 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "sparc-sun-solaris2.10"...
 Reading symbols from /lib/libm.so.2...done.
 Loaded symbols for /lib/libm.so.2
 Reading symbols from /lib/libc.so.1...done.
 Loaded symbols for /lib/libc.so.1
 Reading symbols from /platform/SUNW,A70/lib/libc_psr.so.1...done.
 Loaded symbols for /platform/SUNW,A70/lib/libc_psr.so.1
 Reading symbols from /lib/ld.so.1...done.
 Loaded symbols for /lib/ld.so.1
 Core was generated by `xdelta3 -v -e -s gpl-not.html gpl.html diff.out'.
 Program terminated with signal 11, Segmentation fault.
 #0  0xff1b0c90 in strlen () from /lib/libc.so.1
 (gdb) where
 #0  0xff1b0c90 in strlen () from /lib/libc.so.1
 #1  0xff216258 in _ndoprnt () from /lib/libc.so.1
 #2  0xff2185e4 in sprintf () from /lib/libc.so.1
 #3  0x0002730c in main_format_bcnt (r=24, buf=0x4f8f8 "qu ") at xdelta3-main.h:540
 #4  0x0002cd90 in main_set_source (stream=0x0, cmd=4, sfile=0xffbff040, 
source=0xffbfeb70) at xdelta3-main.h:2153
 #5  0x0002e194 in main_input (cmd=CMD_ENCODE, ifile=0xffbff0b0, ofile=0xffbff078, 
sfile=0xffbff040) at xdelta3-main.h:2503
 #6  0x00030254 in main (argc=2, argv=0xffbff178) at xdelta3-main.h:3221
 (gdb)

3.0t might improve this situation. Please let me know.

Hi Josh,


The program version 3.0t was a lot easier to compile and test. I only
had to define the $CC variable inside the Makefile. However the program
test still failed, as shown below, however my -N workaround no longer
seems to get me past the failure. So I am sticking to the older version.

Please let me know if I can do anything more to help. 

./xdelta3-debug test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=1
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=0
XD3_STDIO=1
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing address_cache (XD3_ALT_CODE_TABLE)... success
xdelta3: testing string_matching...Bus Error (core dumped)


ultra45(fergus)% gdb ./xdelta3-debug core
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.10"...
Reading symbols from /lib/libm.so.2...done.
Loaded symbols for /lib/libm.so.2
Reading symbols from /lib/libc.so.1...done.
Loaded symbols for /lib/libc.so.1
Reading symbols from /platform/SUNW,A70/lib/libc_psr.so.1...done.
Loaded symbols for /platform/SUNW,A70/lib/libc_psr.so.1
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Core was generated by `./xdelta3-debug test'.
Program terminated with signal 10, Bus error.
#0  0x00011ed0 in xd3_string_match_soft (stream=0xffbfd458) at xdelta3.c:5223
5223          if (DO_SMALL) { SMALL_CKSUM_UPDATE (scksum, inp, SLOOK); }
(gdb) quit


ultra45(fergus)% 

I have a post-3.0t that should address this problem -- it was an error found on
ARM/Xscale, and it's fixed but not all tests are passing on that platform. I'll 
post
here when 3.0u is ready.  Thanks.

What is a "post-3.0t" version?
Does this mean there is a version somewhere I download and try?

PS. My email has changed to fergus-at-twig.me.uk, however google seems 
incapable of 
dealing with folk changing email addresses!

Josh,

As per your suggestion I download todays cut (6-aug-2008) from cvs, comnpiled 
and 
ran its test mode. Test mode failed. Here is a log of the results along with 
other 
misc information. Let me know if there is more I can do.

fergus@ultra45: dir
total 1520
drwxr-xr-x   4 fergus   athene2     1024 Aug  6 17:23 ./
drwxrwxr-x   6 fergus   athene2     1024 Aug  6 15:55 ../
-rw-r--r--   1 fergus   athene2     6148 Aug  6 15:16 .DS_Store
-rw-r--r--   1 fergus   athene2       82 Aug  6 15:16 ._.DS_Store
-rw-r--r--   1 fergus   athene2    18009 Feb 11  2007 COPYING
-rw-r--r--   1 fergus   athene2     7705 Aug  6 16:14 Makefile
-rw-r--r--   1 fergus   athene2     6513 Jun 20 05:06 Makefile.mingw
-rw-r--r--   1 fergus   athene2     1067 Nov  9  2007 README
-rw-r--r--   1 fergus   athene2     3600 Feb 11  2007 badcopy.c
-rw-r--r--   1 fergus   athene2     4245 Dec 16  2006 badcopy.vcproj
-rw-r--r--   1 fergus   athene2    60706 Feb 11  2007 draft-korn-vcdiff.txt
drwxr-xr-x   2 fergus   athene2      512 Aug  6 15:11 examples/
-rw-r--r--   1 fergus   athene2     2096 Jul 19 17:16 index.html
-rw-r--r--   1 fergus   athene2     1105 Feb 11  2007 linkxd3lib.c
-rw-r--r--   1 fergus   athene2      255 Mar  6  2007 plot.sh
-rw-rw-r--   1 fergus   athene2      832 Aug  6 17:22 r.r
-rw-r--r--   1 fergus   athene2    36315 Feb 11  2007 rcs_junk.cc
-rw-r--r--   1 fergus   athene2     1067 Mar 25  2007 readme.txt
-rw-r--r--   1 fergus   athene2     2566 Mar 31 04:03 setup.py
drwxr-xr-x   2 fergus   athene2      512 Aug  6 15:13 testing/
-rw-r--r--   1 fergus   athene2     4222 Nov 11  2007 xdelta3-cfgs.h
-rw-r--r--   1 fergus   athene2    29880 Dec 27  2007 xdelta3-decode.h
-rw-r--r--   1 fergus   athene2    47036 Dec 14  2007 xdelta3-djw.h
-rw-r--r--   1 fergus   athene2    22155 Oct 29  2007 xdelta3-fgk.h
-rw-r--r--   1 fergus   athene2     7236 Dec 15  2007 xdelta3-hash.h
-rw-r--r--   1 fergus   athene2     8763 Nov 13  2007 xdelta3-list.h
-rw-r--r--   1 fergus   athene2   107331 Jul  7 04:11 xdelta3-main.h
-rw-r--r--   1 fergus   athene2    11876 Jul  7 04:11 xdelta3-merge.h
-rw-r--r--   1 fergus   athene2     2345 Feb 13  2007 xdelta3-python.h
-rw-r--r--   1 fergus   athene2    33420 Mar 31 04:03 xdelta3-regtest.py
-rw-r--r--   1 fergus   athene2     7172 Dec  2  2007 xdelta3-second.h
-rw-r--r--   1 fergus   athene2    75399 Jul  3 13:59 xdelta3-test.h
-rw-r--r--   1 fergus   athene2     4823 Mar 31 04:03 xdelta3-test.py
-rw-r--r--   1 fergus   athene2   154167 Jul 10 05:10 xdelta3.c
-rw-r--r--   1 fergus   athene2    45508 Jul  3 13:59 xdelta3.h
-rw-r--r--   1 fergus   athene2     2018 Dec  7  2007 xdelta3.prj
-rw-r--r--   1 fergus   athene2     2203 Feb 13  2007 xdelta3.swig
-rw-r--r--   1 fergus   athene2     4938 Mar 26  2007 xdelta3.vcproj
-rw-r--r--   1 fergus   athene2      381 Mar 25  2007 xdelta3.wxi
-rw-r--r--   1 fergus   athene2     4771 Dec  7  2007 xdelta3.wxs
fergus@ultra45: make
gcc -g -Wall -Wshadow -fno-builtin xdelta3.c -lm -o xdelta3-debug \
                -DGENERIC_ENCODE_TABLES=1 \
                -DREGRESSION_TEST=1 \
                -DSECONDARY_DJW=1 \
                -DSECONDARY_FGK=1 \
                -DXD3_DEBUG=1 \
                -DXD3_MAIN=1 \
                -DXD3_STDIO=1 \
                -DXD3_USE_LARGEFILE64=1
gcc -Wall -Wshadow -fno-builtin -O3 xdelta3.c -lm -o xdelta3 \
              -DGENERIC_ENCODE_TABLES=0 \
              -DREGRESSION_TEST=1 \
              -DSECONDARY_DJW=1 \
              -DSECONDARY_FGK=1 \
              -DXD3_DEBUG=0 \
              -DXD3_MAIN=1 \
              -DXD3_POSIX=1 \
              -DXD3_USE_LARGEFILE64=1
fergus@ultra45: ./xdelta3-debug test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=1
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
UNALIGNED_OK=0
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=0
XD3_STDIO=1
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(void*)=4
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... failed: random distribution looks broken: 
XD3_INTERNAL
fergus@ultra45: 
fergus@ultra45: 
fergus@ultra45: uname -a
SunOS ultra45 5.10 Generic_118833-24 sun4u sparc SUNW,A70
fergus@ultra45: gcc --version
gcc (GCC) 3.4.6
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Via e-mail, after the random issue was solved:


xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... 
success
xdelta3: testing
encode_decode_uint64_t..........................................................
.........
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing address_cache (XD3_ALT_CODE_TABLE)... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10
C12/8@qd failed: wrong result: XD3_INTERNAL

This is the sign of a checksum error, related to endianness.  I thought I had 
fixed
this in December of last year, but apparently not.  At this time I do not have a
machine to test this with, but I will investigate.  (I was in the middle of 
working
on Issue 36 and couldn't take my attention away from that, but now I've reached 
a
good stopping point in that work.)

Actually, this looks like an issue with printf(), not endian-ness!

See how it says expected "R/10 C12/8@10" but it got "R0/10 C12/8@qd"?  The "qd" 
is
part of a format specifier for the address portion of the copy instruction, 
which
should be "10" but instead it prints "qd".

I've upgraded from the deprecated "q" specifier to "ll", which I believe is 
portable,
SVN 273.