cea-sec / sibyl Goto Github PK
View Code? Open in Web Editor NEWA Miasm2 based function divination.
License: Other
A Miasm2 based function divination.
License: Other
Hi,
I wrote a simple Binja plugin for using Sibyl.
As of now I'm running it in a single thread but maybe I'll go the way of your plugin and use subprocess
.
Anyway, if you can add a link to it in your repo it would be very nice :)
https://github.com/kenoph/binja_sibyl
(The reason why I did not add it as an ext is that binja requires a plugin.json
file in the root of the repo, along with __init__.py
... I don't want to pollute your repo)
EDIT: Oh yeah, forgot to thank you for the tool. When I read what it does I had an "aha" moment.
As Miasm now uses versioning, it would be fine to have Sibyl following Miasm stable releases, avoiding APIs breaks.
learn
actionHi,
Thank you for your work, this looks amazing :-)
Upon installing Sibyl, I encountered some install errors. The first one it that the python cli tool imports miasm2, which appear to have be renamed to "miasm".
I replaced every occurrence to miasm2 accordingly in Sibyl. And then I have a weird bug:
python2 run_all_tests.py
Traceback (most recent call last):
File "run_all_tests.py", line 6, in <module>
from learn import test_learn
File "/home/vladimir/tools/Sibyl/test/learn/__init__.py", line 1, in <module>
from .run_tests import test_learn
File "/home/vladimir/tools/Sibyl/test/learn/run_tests.py", line 11, in <module>
from sibyl.testlauncher import TestLauncher
File "/usr/lib/python2.7/site-packages/sibyl/testlauncher.py", line 26, in <module>
from sibyl.engine import QEMUEngine, MiasmEngine
File "/usr/lib/python2.7/site-packages/sibyl/engine/__init__.py", line 3, in <module>
from sibyl.engine.qemu import QEMUEngine
File "/usr/lib/python2.7/site-packages/sibyl/engine/qemu.py", line 1, in <module>
from miasm.core.utils import pck32, pck64
ImportError: No module named core.utils
However:
python2 :(
Python 2.7.16 (default, Mar 11 2019, 18:59:25)
[GCC 8.2.1 20181127] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from miasm.core.utils import pck32, pck64
>>>
So, what am I doing wrong? Because I have no idea what the heck is going on.
root@ubuntu:/usr/local/bin# sibyl
Usage: /usr/local/bin/sibyl [action]
Actions:
Traceback (most recent call last):
File "/usr/local/bin/sibyl", line 33, in
key=lambda action: action.name)]
File "/usr/local/bin/sibyl", line 33, in
key=lambda action: action.name)]
AttributeError: 'ActionDesc' object has no attribute 'name'
On latest ubuntu 17.04 x86 and latest miasm
I use function discovering of sibyl cli but it is error:
./sibyl func ~/Downloads/quarkslab.elf
Traceback (most recent call last):
File "./sibyl", line 46, in
load_action(guessed[0], sys.argv[2:])
File "/usr/local/lib/python2.7/dist-packages/sibyl/actions/init.py", line 33, in load_action
return getattr(mod, actiondesc.classname)(args)
File "/usr/local/lib/python2.7/dist-packages/sibyl/actions/action.py", line 36, in init
self.run()
File "/usr/local/lib/python2.7/dist-packages/sibyl/actions/func.py", line 90, in run
for addr in fh.guess():
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/func.py", line 191, in guess
for address, value in self.votes.iteritems():
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/heuristic.py", line 29, in votes
self.do_votes()
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/func.py", line 178, in do_votes
super(FuncHeuristic, self).do_votes()
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/heuristic.py", line 21, in do_votes
for name, vote in heuristic(self).iteritems():
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/func.py", line 102, in pattern_matching
for find_iter, vaddr_base in _virt_find(data, pattern):
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/func.py", line 72, in _virt_find
for s in virt.parent.ph:
AttributeError: 'ELF' object has no attribute 'parent'
It is elfesteem problem? If it true, can you give me a version of elfesteem to fix it. Thanks you
I am interested to work with siby , the most interesting thing is that sibyl can detect the name of the function (strsep , strchr , strncmp ....) , how sibyl can do this ? it has a database of templates and it is just comparing ?
Python>identify_all("x86_32", "ABIStdCall_x86_32")
Launch identification on 3056 function(s)
Current: 65.45% (sub_0x80b6d8c)| Estimated time remaining: 49.80s
0x08057258 : strncmp
0x080576ac : strsep
0x08057b48 : strcmp,strncmp
0x080571cc : strchr
0x08057244 : strlen
0x08057704 : strspn
0x08057b30 : strtok
i tried to work with sibyle , i run this command but it generates this error .
python2 find.py /home/younes/Bureau/a.out ABIStdCall_x86_32 0x400410
Guessed architecture: x86_64
Process Process-4:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "find.py", line 66, in do_test
possible_funcs = tl.run(address, timeout_seconds=timeout)
File "/home/younes/Bureau/Sibyl/sibyl/testlauncher.py", line 181, in run
self.launch_tests(test, address, *args, **kwargs)
File "/home/younes/Bureau/Sibyl/sibyl/testlauncher.py", line 169, in launch_tests
status = test.tests.execute(launch_test)
File "/home/younes/Bureau/Sibyl/sibyl/test/test.py", line 236, in execute
if not callback(init, check):
File "/home/younes/Bureau/Sibyl/sibyl/testlauncher.py", line 144, in launch_test
self.abi.prepare_call(ret_addr=0x1337beef)
File "/home/younes/Bureau/Sibyl/sibyl/abi/abi.py", line 83, in prepare_call
self.vm_push(element)
File "/home/younes/Bureau/Sibyl/sibyl/abi/x86.py", line 33, in vm_push
self.jitter.push_uint32_t(element)
AttributeError: jitter_x86_64 instance has no attribute 'push_uint32_t'
Hi, I faced the question that i can't solve. Yeah, it is a firmware.
First, that i want to start is why sibyl can not even launch test at some addresses? For example, I launch sibyl for one address and after debugging I guessed that launch_tests was not started:
**sibyl find -p -v -t string task 0x103D0
Guessed architecture: arml
Found 1 addresses
Found 3 test cases
1 / 1
Address | Candidates**
After debugging I figured out that, for addresses, which sibyl can work with, in emulation process(qemu), unicorn returned END_ADDR of emulation ( 0x1337babe),
**> /home/vitalym60/.local/lib/python2.7/site-packages/unicorn/unicorn.py(340)reg_read()->322419390L
-> return reg.value
(Pdb) s
--Return--
> /usr/local/lib/python2.7/dist-packages/sibyl/engine/qemu.py(226)__getattr__()->322419390L
-> return self.mu.reg_read(self.pc_reg_value)
(Pdb) getattr(self.cpu, self.ira.pc.name)
322419390L**
and after it, tests are launched.
But for some functions( example above), unicorn return only started address and emulation stopped and tests for guessing wouldn't be started:
if getattr(self.cpu, self.ira.pc.name) != END_ADDR:
raise UnexpectedStopException()
Because of that sibyl can't find function that is there, for example:
IDA shows that at ROM:00182412 memcpy AND after decompiling this function in pseudo C code and building in binary - sibyl detect memcpy.
But now:
**~/Downloads/test sibyl find -p -v -j python -t string -a arml ware.bin 0x00182412
Found 1 addresses
Found 3 test cases
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
1 / 1
Address | Candidates
~/Downloads/test sibyl find -p -v -j qemu -t string -a arml ware.bin 0x00182412
Found 1 addresses
Found 3 test cases
1 / 1
Address | Candidates**
~/Downloads/test sibyl find -p -v -j gcc -t string -a arml ware.bin 0x00182412
Found 1 addresses
Found 3 test cases
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
1 / 1
Address | Candidates
Exactly for this binary, Sibyl can't find any functions.
What the hell is going on?)
Here is the TODO list for the first release (0.1):
find
output: JSON, humanverbose
and quiet
option in action find
SIbyl-tests
find
at end (in verbose mode)doc/
directory with specific documentationHello,
It seems Sibyl (the IDA stub) is unable to recognize the ABI when it finds void functions in IDA.
I believe this issue could be resolved in modifying the gtype_matcher regex as the latter tries to match the keyword before the function arguments.
# int __cdecl(int, int) -> __cdecl
gtype_matcher = re.compile(".+ ([^\(]+)\([^\)]*\)")
Maybe this issue could occur with another keywords. As a reminder, the IDA functions types are defined here: https://www.hex-rays.com/products/ida/support/idadoc/1361.shtml
Hi,
It seems that the IDA script find.py only support IDA7.0 above ! Only IDA7.0 above has the module ida_kernwin!
Dear Sibyl developers,
I am writing regarding an issue I have with a function I am trying to identify.
This function is from a windows driver and imports ntoskrnl.memset
. I have written a stub function to handle it but I could not find a way to use it with the (default) qemu engine.
The problematic block is :
.text:0000000000028ADC mov [rsp+arg_10], rbx
.text:0000000000028AE1 mov [rsp+arg_0], rcx
.text:0000000000028AE6 push rbp
.text:0000000000028AE7 push rsi
.text:0000000000028AE8 push rdi
.text:0000000000028AE9 push r12
.text:0000000000028AEB push r13
.text:0000000000028AED push r14
.text:0000000000028AEF push r15
.text:0000000000028AF1 sub rsp, 100h
.text:0000000000028AF8 mov ebx, edx
.text:0000000000028AFA xor edx, edx ; Val
.text:0000000000028AFC mov rdi, r8
.text:0000000000028AFF mov rsi, rcx
.text:0000000000028B02 lea rcx, [rsp+138h+Dst] ; Dst
.text:0000000000028B0A lea r8d, [rdx+40h] ; Size
.text:0000000000028B0E call memset
.text:0000000000028B13 mov eax, 10h
.text:0000000000028B18 cmp ebx, eax
.text:0000000000028B1A cmovg ebx, eax
.text:0000000000028B1D xor r9d, r9d
.text:0000000000028B20 movsxd rdx, ebx
.text:0000000000028B23 mov r8, r9
.text:0000000000028B26 cmp rdx, r9
.text:0000000000028B29 jle short loc_28B3E
In my sibyl.conf
configuration I added a custom_ntoskrnl.py
stubs file. It contains a function for memset
: a copy/paste from the miasm code for the same function in user-land.
[find]
stubs = /data/miasm/stubs/custom_ntoskrnl.py,/data/miasm/stubs/linux_stdlib.py,/data/ida-scripts/miasm/stubs/win_api_x86_32.py
import logging
from miasm2.jitter.csts import PAGE_READ, PAGE_WRITE, PAGE_EXEC
from miasm2.os_dep.common import heap, windows_to_sbpath
from miasm2.os_dep.common import set_str_unic, set_str_ansi
from miasm2.os_dep.common import get_fmt_args as _get_fmt_args
from miasm2.os_dep.win_api_x86_32_seh import tib_address
def ntoskrnl_memset(jitter):
ret_ad, args = jitter.func_args_cdecl(['addr', 'c', 'size'])
jitter.vm.set_mem(args.addr, chr(args.c) * args.size)
jitter.func_ret_cdecl(ret_ad, args.addr)
With llvm, it works, but not with qemu. From the following extract (generated by using self.jitter.verbose_mode()
:
>>> Tracing instruction at 0x28adc, instruction size = 5
>>> Tracing instruction at 0x28ae1, instruction size = 5
>>> Tracing instruction at 0x28ae6, instruction size = 1
>>> Tracing instruction at 0x28ae7, instruction size = 1
>>> Tracing instruction at 0x28ae8, instruction size = 1
>>> Tracing instruction at 0x28ae9, instruction size = 2
>>> Tracing instruction at 0x28aeb, instruction size = 2
>>> Tracing instruction at 0x28aed, instruction size = 2
>>> Tracing instruction at 0x28aef, instruction size = 2
>>> Tracing instruction at 0x28af1, instruction size = 7
>>> Tracing instruction at 0x28af8, instruction size = 2
>>> Tracing instruction at 0x28afa, instruction size = 2
>>> Tracing instruction at 0x28afc, instruction size = 3
>>> Tracing instruction at 0x28aff, instruction size = 3
>>> Tracing instruction at 0x28b02, instruction size = 8
>>> Tracing instruction at 0x28b0a, instruction size = 4
>>> Tracing instruction at 0x28b0e, instruction size = 5
>>> Tracing instruction at 0x51262, instruction size = 6 // Entered memset after call here
Invalid memory access at 0x0L // Invalid memory access just after
After trying to solve this, I couldn't succeed.
Thank you very much !
Here is the TODO list for the v0.2:
PIN_ROOT
+ tracer path)bzero
implementation to avoid false positivesibyl func
Calling sibyl find
produces an error if pycparser is not installed. Do you think that it could be useful to add a requirements,txt
that will install it?
from the sibyl installation folder on windows I run:
C:\Python27\python.exe find.py d:\path\to\binary -j python -a arml ABI_ARM 0x1768
This gets me:
File "find.py", line 148, in <module>
p.start()
File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
File "C:\Python27\lib\pickle.py", line 636, in _batch_appends
save(tmp[0])
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 396, in save_reduce
save(cls)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thre
ad.lock
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python27\lib\multiprocessing\forking.py", line 380, in main
prepare(preparation_data)
File "C:\Python27\lib\multiprocessing\forking.py", line 495, in prepare
'__parents_main__', file, path_name, etc
File "D:\Sibyl\find.py", line 148, in <module>
p.start()
File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "C:\Python27\lib\multiprocessing\forking.py", line 258, in __init__
cmd = get_command_line() + [rhandle]
File "C:\Python27\lib\multiprocessing\forking.py", line 358, in get_command_l
ine
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.
This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce a Windows executable.
Note that there are two errors (?).
Hi :)
Sibyl by default includes two stubs: win_api_x86_32.py and linux_stdlib.py. Both stubs define different functions with the same name: get_fmt_args. But this breaks some of the stub functions in win_api_x86_32.py that call get_fmt_args, e.g. msvcrt_sprintf with the following error:
TypeError: get_fmt_args() takes exactly 3 arguments (4 given)
To avoid this issue, we have to specify a stub file explicitly.
Hi, am i right that sibyl can't work with emulation at UC_MOD_THUMB (unicorn thumb mode didn't use) ?
Because in "qemu.py" is using only:
uc_mode = unicorn.UC_MODE_ARM + unicorn.UC_MODE_LITTLE_ENDIAN
After installing the latest version of miasm, it seems to break sibyl based off how it tries to import miasm2 when that no longer appears to be a thing (cea-sec/miasm@944806c).
I'm not sure if I'm doing something wrong, perhaps changing all "miasm2" to "miasm" will help?
I was trying to update this to python 3 , but there are some files that were missing when working with the latest version of Miasm.
Yes, I know to work with Miasm stable 1.1 but even that is python2 and it as everyone knows has been dead for a year now.
I was going through slowly updating it until I ran into this
inside
from miasm2.expression.modint import mod_size2int
ModuleNotFoundError: No module named 'miasm.expression.modint'
and indeed there is no file called modint in Miasm. Now this despite being an error of not finding something in MIASM is a problem with Sibyl using/referencing old code.
Here is the TODO list for the v0.3 (non-definitive):
pip
compliantdebug
/explain
action to obtain more information on a given recognition attemptTestIsCharset
, which returns to many false positiveA declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.