noseglasses / elf_diff Goto Github PK

View Code? Open in Web Editor NEW

171.0 171.0 21.0 1.66 MB

A tool to compare ELF binaries

License: GNU General Public License v3.0

Python 54.29% JavaScript 8.95% C++ 1.38% CSS 4.29% HTML 30.06% Shell 0.43% CMake 0.61%

elf_diff's People

Stargazers

Watchers

elf_diff's Issues

Cloning repo issue

Trying to clone this repo in Windows I get the following error:

error: unable to create file python/elf_diff/aux.py: No such file or directory

Warning: Unable to read assembly from binary and the html report has no meaningful data.

Describe the bug
When I compare two files I get an warning.
!!!!!!!!! Warning: Unable to read assembly from binary 'FTD_03_00_00-DEV-BIN/ta1l'. !!!!!!!!!
an the html report has no meaningful data.

To Reproduce
run py -m elf_diff --old_binary_filename "FTD_03_00_00-DEV-BIN/aiu1l" --new_binary_filename "FTD_03_00_00-BIN/aiu1l" --html_file "diff_report.html"

Expected behavior
No warnings or errors and detailed info in the report.

Screenshots
N/A

Desktop (please complete the following information):

OS: Windows 10 (20H2)
Python v3.7.7
elf_diff v0.6.0
objdump, nm, size : (GNU Binutils) v2.32

See attached report and log.
console.log
diff_report.txt

The report is actually the html output.

Exit status does not report "same" or "different", making it impossible to write a script around elf_diff

Describe the bug
I am trying to use elf_diff in a script to compare whether there was a change in code between two compiles of (ostensibly) the same code. Because this is in a script, I was looking to use the exit code from elf_diff (similar to how you can use the exit code from diff, or cmp, to know if the files being compared are the same or have differences). Unfortunately elf_diff appears to always exit with the same exit code regardless of whether there are actual differences.

To Reproduce
Steps to reproduce the behavior:

run comparing a file to itself: elf_diff /bin/bash /bin/bash
Check the output status (in this case, 0)
re-run with files that are definitely different: elf_diff /bin/bash /bin/true
Check the output status and see that it is again 0

Expected behavior
I expected the exit status to be a summary of whether or not the compared files are the same. For example, if you use 'cmp' instead then when comparing /bin/bash to itself you get an exit status of 0, and when comparing bash to true you get an exit status of one. This makes it scriptable.

Run c++filt on the ASM output.

Is your feature request related to a problem? Please describe.

Currently when looking at the assembly output I get the following;

14	mov    %rax,(%rsp)
12	mov    $0xffffffff,%edx
13	xor    %r8d,%r8d
14	call   34 <_ZN3gpl3FFT5doFFTEv+0x34>   30: R_X86_64_PLT32   _ZN3gpl6ddct2dEiiiPPfS0_PiS0_-0x4
15	mov    0xb0(%rbx),%edi
16	test   %edi,%edi
17	jle    72 <_ZN3gpl3FFT5doFFTEv+0x72>

Describe the solution you'd like

If you run c++filt on the above you'll get the following;

14      mov    %rax,(%rsp)
12      mov    bashxffffffff,%edx
13      xor    %r8d,%r8d
14      call   34 <gpl::FFT::doFFT()+0x34>   30: R_X86_64_PLT32   gpl::ddct2d(int, int, int, float**, float*, int*, float*)-0x4
15      mov    0xb0(%rbx),%edi
16      test   %edi,%edi
17      jle    72 <gpl::FFT::doFFT()+0x72>

Which is much nicer to read.

Describe alternatives you've considered

I would have sent a pull request myself but I wasn't able to figure out where to run the c++filt. As a hack I tried adding it to the objdump command in instruction_collector.py:class InstructionCollector(object):gatherSymbolInstructions but it didn't seem to have any effect.

You mention c++filt in the README, so I was surprised that it wasn't already demangling the strings in the assembly output.

Failure if parent of --html_dir does not exist

Describe the bug
If the parent directory of the one specified by --html_dir does not exist, execution fails (see log at the end).
If the directory is created manually beforehand execution works fine.

To Reproduce
Steps to reproduce the behavior:

Make sure the output directory to be specified by --html_dir and its parent does not exist.
Run elf_diff --html_dir parentdir/htmldir ...

Expected behavior
The directory including all leading components should be created.

Desktop (please complete the following information):

OS: Debian 12, fresh venv with only pip and elf_diff (0.7.1) installed (incl. prerequisites)

Additional context

elf_diff.log

Dump structure information from dwarf info

Structure symbol size changes are shown but no way to tell why certain structure size has changed
Thanks for this excellent tool! While the assembly diff for functions helps to explain size differences, when it comes to data structures, there's no extra information in the generated report. Users need to look up the structure manually.

Describe the solution you'd like
Display the structure definition, limited to first level(s) in the information view. This can be done with llvm-dwarfdump <elf> --name=<name> -c The -c lists the children of the default recursive level. Anyway to extract the hierarchy of a structure from the elf file should work.

Pair report generation broken

Describe the bug
The pair report generation fails with an exception.

To Reproduce

 ~ python3 -m elf_diff --bin_prefix arm-none-eabi- --html_file test.html ~/irq_example_c.elf ~/irq_example_cpp.elf 
module_path = /usr/local/lib/python3.9/site-packages/elf_diff
Tools:
   objdump: /usr/local/bin/arm-none-eabi-objdump
   nm:      /usr/local/bin/arm-none-eabi-nm
   size:    /usr/local/bin/arm-none-eabi-size
Symbol selection regex:
   old binary: 'None'
   new binary: 'None'
Symbol exclusion regex:
   old binary: 'None'
   new binary: 'None'
Parsing symbols of old binary (~/irq_example_c.elf)
Parsing symbols of new binary (~/irq_example_cpp.elf)
Symbol Statistics:
   old binary (~/irq_example_c.elf):
      259 total symbol(s)
      259 symbol(s) selected
   new binary (~/irq_example_cpp.elf):
      2 total symbol(s)
      2 symbol(s) selected

   258 persisting symbol(s)
   1 disappeared symbol(s)
   2 new symbol(s)
Analyzing symbol size changes...
100% (258 of 258) |################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Analyzing disappeared symbols...
100% (1 of 1) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Analyzing new symbols...
100% (2 of 2) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Analyzing assembly differences...
100% (258 of 258) |################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Detecting symbol similarities...
100% (1 of 1) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Preparing persisting symbols ...
100% (258 of 258) |################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Preparing disappeared symbols ...
100% (1 of 1) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Preparing new symbols ...
100% (2 of 2) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Preparing similar symbols ...
100% (1 of 1) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/site-packages/elf_diff/__main__.py", line 147, in <module>
    main()
  File "/usr/local/lib/python3.9/site-packages/elf_diff/__main__.py", line 136, in main
    writePairReport(settings)
  File "/usr/local/lib/python3.9/site-packages/elf_diff/__main__.py", line 68, in writePairReport
    single_page_pair_report.writeSinglePageHTMLReport()
  File "/usr/local/lib/python3.9/site-packages/elf_diff/pair_report.py", line 1031, in writeSinglePageHTMLReport
    self.disappeared_symbols_details.getContent()
  File "/usr/local/lib/python3.9/site-packages/elf_diff/pair_report.py", line 64, in getContent
    self.generateContent()
  File "/usr/local/lib/python3.9/site-packages/elf_diff/pair_report.py", line 425, in generateContent
    self.content = f"No functions {self.description}"
AttributeError: 'IsolatedSymbolsDetails' object has no attribute 'description'

Expected behavior
That error should not happen.

Desktop (please complete the following information):

macOS 11.6 w Python 3.9

Additional context
There is a line missing

self.description = description

in here:

elf_diff/src/elf_diff/pair_report.py

Line 419 in fa9014b

def __init__(self, description, isolated_symbols):

Error "Exception: Unnable to find objdump command"

Runtime: Windows7 Python 3.8

CMD: py -m elf_diff diff_note.hmtl app_0210.elf app_0225.elf

Seems need other requirements, I install with pip.

D:\Documents\Elf_Diff_Demo>py -m elf_diff diff_note.hmtl app_0210.elf app_0225.elf

Traceback (most recent call last):
File "D:\Python38\lib\site-packages\elf_diff_main_.py", line 111, in main
settings = Settings(module_path)
File "D:\Python38\lib\site-packages\elf_diff\settings.py", line 353, in init
self._considerCommandLineArgs(cmd_line_args)
File "D:\Python38\lib\site-packages\elf_diff\settings.py", line 508, in _considerCommandLineArgs
self.binutils.initialize(
File "D:\Python38\lib\site-packages\elf_diff\binutils.py", line 106, in initialize
self.findUtility("objdump")
File "D:\Python38\lib\site-packages\elf_diff\binutils.py", line 93, in findUtility
raise Exception(f"Unnable to find {name} command")
Exception: Unnable to find objdump command

elf_diff is unconsolable :-( Something went wrong
Error: Unnable to find objdump command
Don't let this take you down! Have a nice hot coffee and start over.

Utf8 decode error

I am comparing two arm gcc elf files. If I do not specify the bin_dir the report is generated successfully but I get the "Unable to read assembly from binary" warning.

If I specify the correct bin_dir + bin_prefix the warning disappears and instead I get the following output (with utf-8 decode error):

py -m elf_diff --bin_dir tools\arm-gcc\bin --bin_prefix "arm-none-eabi-" --html_dir report2 [OLD].elf [NEW].elf
Tools:
   objdump: tools\arm-gcc\bin\arm-none-eabi-objdump.exe
   nm:      tools\arm-gcc\bin\arm-none-eabi-nm.exe
   readelf:      tools\arm-gcc\bin\arm-none-eabi-readelf.exe
   size:    tools\arm-gcc\bin\arm-none-eabi-size.exe
Verifying config keys...
Symbol selection regex:
   old binary: 'None'
   new binary: 'None'
Symbol exclusion regex:
   old binary: 'None'
   new binary: 'None'
Parsing symbols of old binary ([OLD].elf)
File format of binary [OLD].elf: elf32-littlearm
Extracting symbols
100% (5577 of 5577) |#####################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Gathering instructions
100% (223307 of 223307) |#################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Parsing symbols of new binary ([NEW].elf)
File format of binary [NEW].elf: elf32-littlearm
Extracting symbols
100% (5564 of 5564) |#####################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Gathering instructions
================================================================================

Traceback (most recent call last):
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\__main__.py", line 124, in main
    exportDocument(settings)
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\__main__.py", line 66, in exportDocument
    document: ValueTreeNode = generateDocument(settings)
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\pair_report_document.py", line 1167, in generateDocument
    meta_document.configureValueTree(value_tree, settings=settings)
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\pair_report_document.py", line 976, in configureValueTree
    self.binary_pair = BinaryPair(
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\binary_pair.py", line 103, in __init__
    self.new_binary = Binary(
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\binary.py", line 78, in __init__
    self._initSymbols()
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\binary.py", line 122, in _initSymbols
    self._gatherSymbolInstructions()
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\binary.py", line 108, in _gatherSymbolInstructions
    instruction_collector.gatherSymbolInstructions(
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\instruction_collector.py", line 136, in gatherSymbolInstructions
    objdump_output: str = runSystemCommand(
  File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\system_command.py", line 33, in runSystemCommand
    output: str = o.decode("utf8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 12242097: invalid start byte

================================================================================
 elf_diff is unconsolable :-( Something went wrong
================================================================================

 Error:  'utf-8' codec can't decode byte 0xfc in position 12242097: invalid start byte

================================================================================
 Don't let this take you down! Have a nice hot coffee and start over.
================================================================================

Is there any way I can debug the source of the error / find out what is causing the wrong utf-8 string?

Cannot run `make` from within the _elf_diff_ repo

The README.md tells in ## Setup

The following procedure is required to setup elf_diff.

Install Python version >= 3.0

Clone the elf_diff repo from github.

Run make from within the elf_diff repo or directly install required packages via pip as pip install -r requirements.txt

but I fail in step 3:

$ git clone https://github.com/CapeLeidokos/elf_diff
$ cd elf_diff
$ git remote -v
origin  https://github.com/CapeLeidokos/elf_diff (fetch)
origin  https://github.com/CapeLeidokos/elf_diff (push)
$ make
make: *** No targets specified and no makefile found.  Stop.
$ pip install -r requirements.txt
-bash: pip: command not found
$ python3 --version
Python 3.5.3
$ cat /etc/*release
PRETTY_NAME="Devuan GNU/Linux ascii"
NAME="Devuan GNU/Linux"
ID=devuan 
HOME_URL="https://www.devuan.org/"
SUPPORT_URL="https://devuan.org/os/community"
BUG_REPORT_URL="https://bugs.devuan.org/"

What do I wrongly?

Allow to ignore certain pattern in disassembly diff

Is your feature request related to a problem? Please describe.

The generated diff might show differences on identical code just because other functions where moved. E.g.

is the same code but the disassembled addresses differ because DMAUSBIntHandler was moved some bytes.

Another example is

Due to this, almost all reported differences in "Persisting Symbols" are false positives.

Describe the solution you'd like

It would be nice when certain patterns can be ignored. This list of pattern will depend on the target architecture; e.g. on ARM, running disassembly output through

s!(b[a-z\.]*[[:space:]]+)[[a-f0-9]]*([[:space:]]+<.*>)!\1XXXX\2!

would normalize branch instructions so that only symbol relative information in "<...>" (e.g. DMAUSBIntHandler+0x26) are used for comparision.

Newer gcc labels confuse disassembly listing

I'm trying to analyse my AVR firmware going from gcc-7.3 to gcc-13.2 but the produced diffs are incorrect.

python3 -m elf_diff --bin_prefix "avr-" --html_file gcc7_to_13.html build/avr-gcc-7-variant_at128/MinSizeRel/KateSmib.elf build/avr-gcc-variant_at128/MinSizeRel/KateSmib.elf

avr-gcc 7.3:

00001acc <etl::basic_string_view<char, etl::char_traits<char> >::basic_string_view(char const*)>:
    1acc:	movw	r30, r24
    1ace:	std	Z+1, r23	; 0x01
    1ad0:	st	Z, r22
    1ad2:	ldi	r19, 0x00	; 0
    1ad4:	ldi	r18, 0x00	; 0
    1ad6:	cp	r22, r1
    1ad8:	cpc	r23, r1
    1ada:	breq	.+14     	; 0x1aea <etl::basic_string_view<char, etl::char_traits<char> >::basic_string_view(char const*)+0x1e>
    1adc:	movw	r30, r22
    1ade:	movw	r18, r30
    1ae0:	sub	r18, r22
    1ae2:	sbc	r19, r23
    1ae4:	ld	r20, Z+
    1ae6:	cpse	r20, r1
    1ae8:	rjmp	.-12     	; 0x1ade <etl::basic_string_view<char, etl::char_traits<char> >::basic_string_view(char const*)+0x12>
    1aea:	add	r22, r18
    1aec:	adc	r23, r19
    1aee:	movw	r30, r24
    1af0:	std	Z+3, r23	; 0x03
    1af2:	std	Z+2, r22	; 0x02
    1af4:	ret

avr-gcc 13.2:

00001e0e <etl::basic_string_view<char, etl::char_traits<char> >::basic_string_view(char const*)>:
    ETL_CONSTEXPR14  ETL_EXPLICIT_STRING_FROM_CHAR basic_string_view(const T* begin_)
    1e0e:	movw	r26, r24
    1e10:	movw	r24, r22

00001e12 <.Loc.778>:
      : mbegin(begin_)
    1e12:	adiw	r26, 0x01	; 1
    1e14:	st	X, r23
    1e16:	st	-X, r22

00001e18 <.LBB6103>:
    }

    //*************************************************************************
    static ETL_CONSTEXPR14 size_t length(const char_type* str)
    {
      size_t count = 0UL;
    1e18:	ldi	r18, 0x00	; 0
    1e1a:	ldi	r19, 0x00	; 0

00001e1c <.Loc.781>:

      if (str != 0)
    1e1c:	sbiw	r24, 0x00	; 0
    1e1e:	breq	.+28     	; 0x1e3c <.L158>
    1e20:	mov	r21, r22
    1e22:	mov	r20, r25

00001e24 <.LBB6105>:
      size_t count = 0UL;
    1e24:	ldi	r18, 0x00	; 0
    1e26:	ldi	r19, 0x00	; 0
    1e28:	rjmp	.+8      	; 0x1e32 <.L157>

00001e2a <.L159>:
      {
        while (*str++ != 0)
        {
          ++count;
    1e2a:	subi	r18, 0xFF	; 255
    1e2c:	sbci	r19, 0xFF	; 255

00001e2e <.Loc.784>:
        while (*str++ != 0)
    1e2e:	mov	r21, r30
    1e30:	mov	r20, r31

00001e32 <.L157>:
    1e32:	mov	r30, r21
    1e34:	mov	r31, r20

00001e36 <.Loc.786>:
    1e36:	ld	r20, Z+

00001e38 <.Loc.787>:
    1e38:	cpse	r20, r1
    1e3a:	rjmp	.-18     	; 0x1e2a <.L159>

00001e3c <.L158>:
      , mend(begin_ + TTraits::length(begin_))
    1e3c:	add	r24, r18
    1e3e:	adc	r25, r19

00001e40 <.Loc.790>:
    1e40:	adiw	r26, 0x03	; 3
    1e42:	st	X, r25
    1e44:	st	-X, r24
    1e46:	sbiw	r26, 0x02	; 2

00001e48 <.Loc.791>:
    }
    1e48:	ret

Bug in handling migrated symbols

Describe the bug
I have an embedded project for an ARM microcontroller that I need to be able to compile with the vendor's Eclipse environment (that generates GNU make makefiles) and a CMake environment. Both use the same gcc-based toolchain binaries (as of now based on GCC 12.3.1). I am using elf_diff to ensure that the resulting binaries are equal "enough". This worked fine so far with different configurations (e.g., different linker scripts, applications).

Eventually, I ran into a false positive (reporting the files to differ although they not AFAICT) that also shows a wrong location to the respective symbols for one of the ELF files.

To Reproduce
It's not exactly easy to provide a MWE including the sources (and assuming this is ARM-specific you would need the right toolchain too). The main culprit is this function. I could provide you the ELF files though (I'd rather do that in private though because this is work-related and contains the customer and project name in the paths :).
I execute elf_diff with --skip_symbol_similarities, --bin_dir pointing to the ARM toolchain used for building, and --bin_prefix "arm-none-eabi-".

I found out some interesting and hopefully at least partially helpful facts:

Both ELF files work fine in practice as far as execution is concerned.
The order of the object files during linking is important. I can make the false positive go away by swapping two files around in the linker's command line(!).
Neither of the two object files involved in the swapping contain the symbols reported in the false positive.
The respective function is an interrupt handler function that is defined with __attribute__ ((weak, section(".after_vectors"))) and has a declared prototype with __attribute__ ((weak)) only. This function is than aliased to 134 other function names with __attribute__ ((weak, alias (...))).
The multipage html output correctly lists all of the function names at the same line where the actual definition is located for one of the ELF files and consistently on a wrong line and wrong file for the other ELF file.
Dumping the debug info (with arm-none-eabi-readelf -w) shows a lot of warnings for both sides including numerous readelf: Warning: There is a hole [... - ...] in .debug_loc section. and exactly 10 occurrences of readelf: Warning: Hole and overlap detection requires adjacent view lists and loclists. each. (I don't know why 10 times yet. There are 22 object files involved).
This happens only for one linker script configuration where the function in question is mapped to a physical address near 0 (namely to 0x000002ee).

I couldn't find out what the wrong side is actually pointing to. As I mentioned the file pointed to does not contain any of the affected symbols at all. And the line number is also different but I could not determine where it is coming from. From all of the above, I think this is either a bug in ld or the ELF parsing (or both) that is triggered by some peculiar debug info of aliased functions.

Expected behavior
I am not sure exactly. Ideally, the expected behavior should be that it just works and shows the files to be equal. Alternatively, it could probably also try to detect the erroneous circumstance and report this as an error.

Screenshots

Desktop (please complete the following information):

OS: Debian stable
Version 0.7.1 from pip

AttributeError: 'Binary' object has no attribute 'progmem_size'

Seem like a great utility, unfortunately unable to use due to this error:

elf_diff --old one.elf  --new two.elf  
bin_path = .../elf_diff/bin
bin_path = .../elf_diff/bin
Traceback (most recent call last):
  File ".../elf_diff/bin/elf_diff", line 52, in <module>
    generatePairReport(settings)
  File ".../elf_diff/bin/../python/elf_diff/pair_report.py", line 405, in generatePairReport
    PairReport(settings).generate()
  File ".../elf_diff/bin/../python/elf_diff/pair_report.py", line 38, in __init__
    settings.new_binary_filename)
  File ".../elf_diff/bin/../python/elf_diff/binary_pair.py", line 41, in __init__
    self.old_binary = Binary(self.settings, self.old_binary_filename)
  File ".../elf_diff/bin/../python/elf_diff/binary.py", line 44, in __init__
    self.parseSymbols()
  File ".../elf_diff/bin/../python/elf_diff/binary.py", line 139, in parseSymbols
    cur_symbol.addInstructions(instruction_line_match.group(1) + instruction_line_match.group(2))
AttributeError: 'NoneType' object has no attribute 'addInstructions'

conda list:

# Name                    Version                   Build  Channel
ca-certificates           2020.10.14                    0  
certifi                   2019.11.28               py27_0  
jinja2                    2.11.2                   pypi_0    pypi
libcxx                    10.0.0                        1  
libedit                   3.1.20191231         h1de35cc_1  
libffi                    3.3                  hb1e8313_2  
markupsafe                1.1.1                    pypi_0    pypi
ncurses                   6.2                  h0a44026_1  
pdfkit                    0.6.1                    pypi_0    pypi
pip                       19.3.1                   py27_0  
python                    2.7.18               h47d645e_1  
pyyaml                    5.3.1                    pypi_0    pypi
readline                  8.0                  h1de35cc_0  
setuptools                44.0.0                   py27_0  
sqlite                    3.33.0               hffcf06c_0  
tk                        8.6.10               hb0a8c7a_0  
wheel                     0.35.1                     py_0  
zlib                      1.2.11               h1de35cc_3

noseglasses / elf_diff Goto Github PK

elf_diff's People

Stargazers

Watchers

Forkers

elf_diff's Issues

Recommend Projects

Recommend Topics

Recommend Org