noseglasses / elf_diff Goto Github PK
View Code? Open in Web Editor NEWA tool to compare ELF binaries
License: GNU General Public License v3.0
A tool to compare ELF binaries
License: GNU General Public License v3.0
Trying to clone this repo in Windows I get the following error:
error: unable to create file python/elf_diff/aux.py: No such file or directory
foo
bar
Describe the bug
When I compare two files I get an warning.
!!!!!!!!! Warning: Unable to read assembly from binary 'FTD_03_00_00-DEV-BIN/ta1l'. !!!!!!!!!
an the html report has no meaningful data.
To Reproduce
run py -m elf_diff --old_binary_filename "FTD_03_00_00-DEV-BIN/aiu1l" --new_binary_filename "FTD_03_00_00-BIN/aiu1l" --html_file "diff_report.html"
Expected behavior
No warnings or errors and detailed info in the report.
Screenshots
N/A
Desktop (please complete the following information):
See attached report and log.
console.log
diff_report.txt
The report is actually the html output.
Describe the bug
I am trying to use elf_diff in a script to compare whether there was a change in code between two compiles of (ostensibly) the same code. Because this is in a script, I was looking to use the exit code from elf_diff (similar to how you can use the exit code from diff, or cmp, to know if the files being compared are the same or have differences). Unfortunately elf_diff appears to always exit with the same exit code regardless of whether there are actual differences.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
I expected the exit status to be a summary of whether or not the compared files are the same. For example, if you use 'cmp' instead then when comparing /bin/bash to itself you get an exit status of 0, and when comparing bash to true you get an exit status of one. This makes it scriptable.
Is your feature request related to a problem? Please describe.
Currently when looking at the assembly output I get the following;
14 mov %rax,(%rsp)
12 mov $0xffffffff,%edx
13 xor %r8d,%r8d
14 call 34 <_ZN3gpl3FFT5doFFTEv+0x34> 30: R_X86_64_PLT32 _ZN3gpl6ddct2dEiiiPPfS0_PiS0_-0x4
15 mov 0xb0(%rbx),%edi
16 test %edi,%edi
17 jle 72 <_ZN3gpl3FFT5doFFTEv+0x72>
Describe the solution you'd like
If you run c++filt
on the above you'll get the following;
14 mov %rax,(%rsp)
12 mov bashxffffffff,%edx
13 xor %r8d,%r8d
14 call 34 <gpl::FFT::doFFT()+0x34> 30: R_X86_64_PLT32 gpl::ddct2d(int, int, int, float**, float*, int*, float*)-0x4
15 mov 0xb0(%rbx),%edi
16 test %edi,%edi
17 jle 72 <gpl::FFT::doFFT()+0x72>
Which is much nicer to read.
Describe alternatives you've considered
I would have sent a pull request myself but I wasn't able to figure out where to run the c++filt
. As a hack I tried adding it to the objdump
command in instruction_collector.py:class InstructionCollector(object):gatherSymbolInstructions
but it didn't seem to have any effect.
You mention c++filt
in the README, so I was surprised that it wasn't already demangling the strings in the assembly output.
Describe the bug
If the parent directory of the one specified by --html_dir
does not exist, execution fails (see log at the end).
If the directory is created manually beforehand execution works fine.
To Reproduce
Steps to reproduce the behavior:
--html_dir
and its parent does not exist.elf_diff --html_dir parentdir/htmldir ...
Expected behavior
The directory including all leading components should be created.
Desktop (please complete the following information):
Additional context
Structure symbol size changes are shown but no way to tell why certain structure size has changed
Thanks for this excellent tool! While the assembly diff for functions helps to explain size differences, when it comes to data structures, there's no extra information in the generated report. Users need to look up the structure manually.
Describe the solution you'd like
Display the structure definition, limited to first level(s) in the information view. This can be done with llvm-dwarfdump <elf> --name=<name> -c
The -c
lists the children of the default recursive level. Anyway to extract the hierarchy of a structure from the elf file should work.
Describe the bug
The pair report generation fails with an exception.
To Reproduce
~ python3 -m elf_diff --bin_prefix arm-none-eabi- --html_file test.html ~/irq_example_c.elf ~/irq_example_cpp.elf
module_path = /usr/local/lib/python3.9/site-packages/elf_diff
Tools:
objdump: /usr/local/bin/arm-none-eabi-objdump
nm: /usr/local/bin/arm-none-eabi-nm
size: /usr/local/bin/arm-none-eabi-size
Symbol selection regex:
old binary: 'None'
new binary: 'None'
Symbol exclusion regex:
old binary: 'None'
new binary: 'None'
Parsing symbols of old binary (~/irq_example_c.elf)
Parsing symbols of new binary (~/irq_example_cpp.elf)
Symbol Statistics:
old binary (~/irq_example_c.elf):
259 total symbol(s)
259 symbol(s) selected
new binary (~/irq_example_cpp.elf):
2 total symbol(s)
2 symbol(s) selected
258 persisting symbol(s)
1 disappeared symbol(s)
2 new symbol(s)
Analyzing symbol size changes...
100% (258 of 258) |################################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Analyzing disappeared symbols...
100% (1 of 1) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Analyzing new symbols...
100% (2 of 2) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Analyzing assembly differences...
100% (258 of 258) |################################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Detecting symbol similarities...
100% (1 of 1) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Preparing persisting symbols ...
100% (258 of 258) |################################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Preparing disappeared symbols ...
100% (1 of 1) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Preparing new symbols ...
100% (2 of 2) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Preparing similar symbols ...
100% (1 of 1) |####################################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Traceback (most recent call last):
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.9/site-packages/elf_diff/__main__.py", line 147, in <module>
main()
File "/usr/local/lib/python3.9/site-packages/elf_diff/__main__.py", line 136, in main
writePairReport(settings)
File "/usr/local/lib/python3.9/site-packages/elf_diff/__main__.py", line 68, in writePairReport
single_page_pair_report.writeSinglePageHTMLReport()
File "/usr/local/lib/python3.9/site-packages/elf_diff/pair_report.py", line 1031, in writeSinglePageHTMLReport
self.disappeared_symbols_details.getContent()
File "/usr/local/lib/python3.9/site-packages/elf_diff/pair_report.py", line 64, in getContent
self.generateContent()
File "/usr/local/lib/python3.9/site-packages/elf_diff/pair_report.py", line 425, in generateContent
self.content = f"No functions {self.description}"
AttributeError: 'IsolatedSymbolsDetails' object has no attribute 'description'
Expected behavior
That error should not happen.
Desktop (please complete the following information):
Additional context
There is a line missing
self.description = description
in here:
elf_diff/src/elf_diff/pair_report.py
Line 419 in fa9014b
Runtime: Windows7 Python 3.8
CMD: py -m elf_diff diff_note.hmtl app_0210.elf app_0225.elf
Seems need other requirements, I install with pip.
D:\Documents\Elf_Diff_Demo>py -m elf_diff diff_note.hmtl app_0210.elf app_0225.elf
Traceback (most recent call last):
File "D:\Python38\lib\site-packages\elf_diff_main_.py", line 111, in main
settings = Settings(module_path)
File "D:\Python38\lib\site-packages\elf_diff\settings.py", line 353, in init
self._considerCommandLineArgs(cmd_line_args)
File "D:\Python38\lib\site-packages\elf_diff\settings.py", line 508, in _considerCommandLineArgs
self.binutils.initialize(
File "D:\Python38\lib\site-packages\elf_diff\binutils.py", line 106, in initialize
self.findUtility("objdump")
File "D:\Python38\lib\site-packages\elf_diff\binutils.py", line 93, in findUtility
raise Exception(f"Unnable to find {name} command")
Exception: Unnable to find objdump commandelf_diff is unconsolable :-( Something went wrong
Error: Unnable to find objdump command
Don't let this take you down! Have a nice hot coffee and start over.
I am comparing two arm gcc elf files. If I do not specify the bin_dir the report is generated successfully but I get the "Unable to read assembly from binary" warning.
If I specify the correct bin_dir + bin_prefix the warning disappears and instead I get the following output (with utf-8 decode error):
py -m elf_diff --bin_dir tools\arm-gcc\bin --bin_prefix "arm-none-eabi-" --html_dir report2 [OLD].elf [NEW].elf
Tools:
objdump: tools\arm-gcc\bin\arm-none-eabi-objdump.exe
nm: tools\arm-gcc\bin\arm-none-eabi-nm.exe
readelf: tools\arm-gcc\bin\arm-none-eabi-readelf.exe
size: tools\arm-gcc\bin\arm-none-eabi-size.exe
Verifying config keys...
Symbol selection regex:
old binary: 'None'
new binary: 'None'
Symbol exclusion regex:
old binary: 'None'
new binary: 'None'
Parsing symbols of old binary ([OLD].elf)
File format of binary [OLD].elf: elf32-littlearm
Extracting symbols
100% (5577 of 5577) |#####################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Gathering instructions
100% (223307 of 223307) |#################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Parsing symbols of new binary ([NEW].elf)
File format of binary [NEW].elf: elf32-littlearm
Extracting symbols
100% (5564 of 5564) |#####################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
Gathering instructions
================================================================================
Traceback (most recent call last):
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\__main__.py", line 124, in main
exportDocument(settings)
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\__main__.py", line 66, in exportDocument
document: ValueTreeNode = generateDocument(settings)
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\pair_report_document.py", line 1167, in generateDocument
meta_document.configureValueTree(value_tree, settings=settings)
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\pair_report_document.py", line 976, in configureValueTree
self.binary_pair = BinaryPair(
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\binary_pair.py", line 103, in __init__
self.new_binary = Binary(
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\binary.py", line 78, in __init__
self._initSymbols()
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\binary.py", line 122, in _initSymbols
self._gatherSymbolInstructions()
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\binary.py", line 108, in _gatherSymbolInstructions
instruction_collector.gatherSymbolInstructions(
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\instruction_collector.py", line 136, in gatherSymbolInstructions
objdump_output: str = runSystemCommand(
File "C:\[...]\Python\Python310\lib\site-packages\elf_diff\system_command.py", line 33, in runSystemCommand
output: str = o.decode("utf8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 12242097: invalid start byte
================================================================================
elf_diff is unconsolable :-( Something went wrong
================================================================================
Error: 'utf-8' codec can't decode byte 0xfc in position 12242097: invalid start byte
================================================================================
Don't let this take you down! Have a nice hot coffee and start over.
================================================================================
Is there any way I can debug the source of the error / find out what is causing the wrong utf-8 string?
The README.md tells in ## Setup
The following procedure is required to setup elf_diff.
- Install Python version >= 3.0
- Clone the elf_diff repo from github.
- Run
make
from within the elf_diff repo or directly install required packages via pip aspip install -r requirements.txt
but I fail in step 3:
$ git clone https://github.com/CapeLeidokos/elf_diff
$ cd elf_diff
$ git remote -v
origin https://github.com/CapeLeidokos/elf_diff (fetch)
origin https://github.com/CapeLeidokos/elf_diff (push)
$ make
make: *** No targets specified and no makefile found. Stop.
$ pip install -r requirements.txt
-bash: pip: command not found
$ python3 --version
Python 3.5.3
$ cat /etc/*release
PRETTY_NAME="Devuan GNU/Linux ascii"
NAME="Devuan GNU/Linux"
ID=devuan
HOME_URL="https://www.devuan.org/"
SUPPORT_URL="https://devuan.org/os/community"
BUG_REPORT_URL="https://bugs.devuan.org/"
What do I wrongly?
Is your feature request related to a problem? Please describe.
The generated diff might show differences on identical code just because other functions where moved. E.g.
is the same code but the disassembled addresses differ because DMAUSBIntHandler
was moved some bytes.
Another example is
Due to this, almost all reported differences in "Persisting Symbols" are false positives.
Describe the solution you'd like
It would be nice when certain patterns can be ignored. This list of pattern will depend on the target architecture; e.g. on ARM, running disassembly output through
s!(b[a-z\.]*[[:space:]]+)[[a-f0-9]]*([[:space:]]+<.*>)!\1XXXX\2!
would normalize branch instructions so that only symbol relative information in "<...>" (e.g. DMAUSBIntHandler+0x26
) are used for comparision.
I'm trying to analyse my AVR firmware going from gcc-7.3 to gcc-13.2 but the produced diffs are incorrect.
python3 -m elf_diff --bin_prefix "avr-" --html_file gcc7_to_13.html build/avr-gcc-7-variant_at128/MinSizeRel/KateSmib.elf build/avr-gcc-variant_at128/MinSizeRel/KateSmib.elf
avr-gcc 7.3:
00001acc <etl::basic_string_view<char, etl::char_traits<char> >::basic_string_view(char const*)>:
1acc: movw r30, r24
1ace: std Z+1, r23 ; 0x01
1ad0: st Z, r22
1ad2: ldi r19, 0x00 ; 0
1ad4: ldi r18, 0x00 ; 0
1ad6: cp r22, r1
1ad8: cpc r23, r1
1ada: breq .+14 ; 0x1aea <etl::basic_string_view<char, etl::char_traits<char> >::basic_string_view(char const*)+0x1e>
1adc: movw r30, r22
1ade: movw r18, r30
1ae0: sub r18, r22
1ae2: sbc r19, r23
1ae4: ld r20, Z+
1ae6: cpse r20, r1
1ae8: rjmp .-12 ; 0x1ade <etl::basic_string_view<char, etl::char_traits<char> >::basic_string_view(char const*)+0x12>
1aea: add r22, r18
1aec: adc r23, r19
1aee: movw r30, r24
1af0: std Z+3, r23 ; 0x03
1af2: std Z+2, r22 ; 0x02
1af4: ret
avr-gcc 13.2:
00001e0e <etl::basic_string_view<char, etl::char_traits<char> >::basic_string_view(char const*)>:
ETL_CONSTEXPR14 ETL_EXPLICIT_STRING_FROM_CHAR basic_string_view(const T* begin_)
1e0e: movw r26, r24
1e10: movw r24, r22
00001e12 <.Loc.778>:
: mbegin(begin_)
1e12: adiw r26, 0x01 ; 1
1e14: st X, r23
1e16: st -X, r22
00001e18 <.LBB6103>:
}
//*************************************************************************
static ETL_CONSTEXPR14 size_t length(const char_type* str)
{
size_t count = 0UL;
1e18: ldi r18, 0x00 ; 0
1e1a: ldi r19, 0x00 ; 0
00001e1c <.Loc.781>:
if (str != 0)
1e1c: sbiw r24, 0x00 ; 0
1e1e: breq .+28 ; 0x1e3c <.L158>
1e20: mov r21, r22
1e22: mov r20, r25
00001e24 <.LBB6105>:
size_t count = 0UL;
1e24: ldi r18, 0x00 ; 0
1e26: ldi r19, 0x00 ; 0
1e28: rjmp .+8 ; 0x1e32 <.L157>
00001e2a <.L159>:
{
while (*str++ != 0)
{
++count;
1e2a: subi r18, 0xFF ; 255
1e2c: sbci r19, 0xFF ; 255
00001e2e <.Loc.784>:
while (*str++ != 0)
1e2e: mov r21, r30
1e30: mov r20, r31
00001e32 <.L157>:
1e32: mov r30, r21
1e34: mov r31, r20
00001e36 <.Loc.786>:
1e36: ld r20, Z+
00001e38 <.Loc.787>:
1e38: cpse r20, r1
1e3a: rjmp .-18 ; 0x1e2a <.L159>
00001e3c <.L158>:
, mend(begin_ + TTraits::length(begin_))
1e3c: add r24, r18
1e3e: adc r25, r19
00001e40 <.Loc.790>:
1e40: adiw r26, 0x03 ; 3
1e42: st X, r25
1e44: st -X, r24
1e46: sbiw r26, 0x02 ; 2
00001e48 <.Loc.791>:
}
1e48: ret
Describe the bug
I have an embedded project for an ARM microcontroller that I need to be able to compile with the vendor's Eclipse environment (that generates GNU make makefiles) and a CMake environment. Both use the same gcc-based toolchain binaries (as of now based on GCC 12.3.1). I am using elf_diff to ensure that the resulting binaries are equal "enough". This worked fine so far with different configurations (e.g., different linker scripts, applications).
Eventually, I ran into a false positive (reporting the files to differ although they not AFAICT) that also shows a wrong location to the respective symbols for one of the ELF files.
To Reproduce
It's not exactly easy to provide a MWE including the sources (and assuming this is ARM-specific you would need the right toolchain too). The main culprit is this function. I could provide you the ELF files though (I'd rather do that in private though because this is work-related and contains the customer and project name in the paths :).
I execute elf_diff with --skip_symbol_similarities
, --bin_dir
pointing to the ARM toolchain used for building, and --bin_prefix "arm-none-eabi-"
.
I found out some interesting and hopefully at least partially helpful facts:
__attribute__ ((weak, section(".after_vectors")))
and has a declared prototype with __attribute__ ((weak))
only. This function is than aliased to 134 other function names with __attribute__ ((weak, alias (...)))
.arm-none-eabi-readelf -w
) shows a lot of warnings for both sides including numerous readelf: Warning: There is a hole [... - ...] in .debug_loc section.
and exactly 10 occurrences of readelf: Warning: Hole and overlap detection requires adjacent view lists and loclists.
each. (I don't know why 10 times yet. There are 22 object files involved).0x000002ee
).I couldn't find out what the wrong side is actually pointing to. As I mentioned the file pointed to does not contain any of the affected symbols at all. And the line number is also different but I could not determine where it is coming from. From all of the above, I think this is either a bug in ld
or the ELF parsing (or both) that is triggered by some peculiar debug info of aliased functions.
Expected behavior
I am not sure exactly. Ideally, the expected behavior should be that it just works and shows the files to be equal. Alternatively, it could probably also try to detect the erroneous circumstance and report this as an error.
Desktop (please complete the following information):
Seem like a great utility, unfortunately unable to use due to this error:
elf_diff --old one.elf --new two.elf
bin_path = .../elf_diff/bin
bin_path = .../elf_diff/bin
Traceback (most recent call last):
File ".../elf_diff/bin/elf_diff", line 52, in <module>
generatePairReport(settings)
File ".../elf_diff/bin/../python/elf_diff/pair_report.py", line 405, in generatePairReport
PairReport(settings).generate()
File ".../elf_diff/bin/../python/elf_diff/pair_report.py", line 38, in __init__
settings.new_binary_filename)
File ".../elf_diff/bin/../python/elf_diff/binary_pair.py", line 41, in __init__
self.old_binary = Binary(self.settings, self.old_binary_filename)
File ".../elf_diff/bin/../python/elf_diff/binary.py", line 44, in __init__
self.parseSymbols()
File ".../elf_diff/bin/../python/elf_diff/binary.py", line 139, in parseSymbols
cur_symbol.addInstructions(instruction_line_match.group(1) + instruction_line_match.group(2))
AttributeError: 'NoneType' object has no attribute 'addInstructions'
conda list:
# Name Version Build Channel
ca-certificates 2020.10.14 0
certifi 2019.11.28 py27_0
jinja2 2.11.2 pypi_0 pypi
libcxx 10.0.0 1
libedit 3.1.20191231 h1de35cc_1
libffi 3.3 hb1e8313_2
markupsafe 1.1.1 pypi_0 pypi
ncurses 6.2 h0a44026_1
pdfkit 0.6.1 pypi_0 pypi
pip 19.3.1 py27_0
python 2.7.18 h47d645e_1
pyyaml 5.3.1 pypi_0 pypi
readline 8.0 h1de35cc_0
setuptools 44.0.0 py27_0
sqlite 3.33.0 hffcf06c_0
tk 8.6.10 hb0a8c7a_0
wheel 0.35.1 py_0
zlib 1.2.11 h1de35cc_3
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.