Please some more documentation, especially about the IDAAPI

It seems that addresses beyond 0x80000000 are interpreted as negative numbers, even on a 64-bit python3. The address -2147360092 or -0x7ffe1d5c is actually 0x8001E2A4 which is a valid address:

./ScratchABit.py EXT0CB6Q.dec.P2.def --script import
Loading CPU plugin arm_32_capstone
Adding area: Area(0x0-0x1ffff, {'name': '.bin', 'access': 'RWX'})
Adding area: Area(0x808000-0x80ffff, {'name': '.bin', 'access': 'RWX'})
Adding area: Area(0x80000200-0x800301ff, {'name': '.bin', 'access': 'RWX'})
Loading EXT0CB6Q.dec.P21.frmw @0x0
Loading EXT0CB6Q.dec.P22.frmw @0x808000
Loading EXT0CB6Q.dec.P23.frmw @0x80000200
Processing section: entrypoints
Performing initial analysis... 3000
Traceback (most recent call last):
File "./ScratchABit.py", line 1030, in
call_script(script)
File "./ScratchABit.py", line 949, in call_script
mod = import(script)
File "/home/user/scratch/ScratchABit/import.py", line 2464, in
SetRegEx(0x8000855A,"T",1,2)
File "/home/user/scratch/ScratchABit/idc.py", line 65, in SetRegEx
engine.analyze()
File "/home/user/scratch/ScratchABit/scratchabit/engine.py", line 947, in analyze
if not _processor.emu():
File "/home/user/scratch/ScratchABit/plugins/cpu/_any_capstone.py", line 176, in emu
assert GetReg(op.addr, "T") == 0
File "/home/user/scratch/ScratchABit/idc.py", line 28, in GetReg
if engine.ADDRESS_SPACE.get_flags(ea, engine.AddressSpace.ALT_CODE):
File "/home/user/scratch/ScratchABit/scratchabit/engine.py", line 245, in get_flags
raise InvalidAddrException(addr)
scratchabit.defs.InvalidAddrException: (-2147360092, '-0x7ffe1d5c')

https://github.com/williballenthin/python-idb is advertized as a tool to decode IDB databases, but that's boring part, it actually had 50K of the code implementing idaapi and friends.

Would be awesome if ScratchABit would support the ESP32 in the future...

Please add documentation about the various CPU plugins, what they do, and what their limitations are, so that a user can know which plugin to choose.

Like "lock", etc.

Hi,

I'm having trouble running ScratchABit on my machine:

$ python scratchabit.py example-elf
Traceback (most recent call last):
  File "scratchabit.py", line 721, in <module>
    load_target_file(loader, sys.argv[1])
  File "scratchabit.py", line 623, in load_target_file
    entry = loader.load(engine.ADDRESS_SPACE, fname)
  File "/home/user/esp/virtenv/ScratchABit/plugins/loader/elf.py", line 407, in load
    return load_sections(aspace, elffile)
  File "/home/user/esp/virtenv/ScratchABit/plugins/loader/elf.py", line 280, in load_sections
    if reloc["r_addend"] != 0:
  File "/home/user/esp/virtenv/ScratchABit/plugins/loader/pyelftools/elftools/elf/relocation.py", line 35, in __getitem__
    return self.entry[name]
  File "/home/user/esp/virtenv/ScratchABit/plugins/loader/pyelftools/elftools/construct/lib/container.py", line 35, in __getitem__
    return self.__dict__[name]
KeyError: 'r_addend'

Am I missing something?
I use Ubuntu 14.04 with Python 3.4.3
Thanks

Example of disasm:

│08020036          nlr_jump:                                                                                                           │
│08020036 0d4b         ldr      r3, [pc, #52]                                                                                          │
│08020038 1a68         ldr      r2, [r3, #0]                                                                                           │
│0802003a 002a         cmp      r2, #0                                                                                                 │
│0802003c 0df0de83     b.w      nlr_jump_fail                                                                                          │
│08020040 50           unk     0x50 ; 'P'                                                                                              │
│08020041 60           unk     0x60 ; '`'                                                                                              │
│08020042 10           unk     0x10                                                                                                    │
│08020043 68           unk     0x68 ; 'h'                                                                                              │
│08020044 18           unk     0x18                                                                                                    │
│08020045 60           unk     0x60 ; '`'                                                 

That b.w can't be right, should be conditional.

(Disassembling MicroPython's bare-arm port.)

Quoting the README:

See how to support other types of IDAPython plugins besides just processor modules.

Loader plugins would be highly appreciated. The problem is that Python loaders are hard to find. The existing loaders, covering widespread binary executable formats, are usually written in C++ so they need to be ported to IDAPython first.
Fortunately, stand-alone Python libraries for parsing several common executable formats exist:

• pefile for parsing M$ PE files
• macholib for Mach-O executables

Creating basic loader plugins ontop of them should be easy. The existing pyelftools based ELF loader serves as a good example of how to do that.
The first step is to enable support for loader plugins in ScratchABit...

ScratchABit generally works well for typical "embedded" size of projects, on the order of 1MB of raw code. It's also written in a way "aware" of potential bottlenecks, e.g. common operations, like browsing around and DB saving are optimized, operation which are known to be "not instant", e.g. complete disassembly write-out or full-text search, have progress indicators.

However, would be useful to perform more dedicated load testing on bigger files. Following areas can be identified:

1. Opening of structure executable/object formats (currently ELF is supported). For example, parsing ELF files with big string tables may take noticeable time (handled by 3rd-part lib). E.g., would be nice to test performance of loading tens-of-MB ELF files.
2. Testing saving of projects with 8MB or more of raw binary code.
3. Like above, but disassembly writing/full-text search.

This essentially means that different functions share some basic blocks, and gives good hint how to deal with that - there should be multiple functions, but with overlapping ranges. SAB already exports functions based on ranges, so that part is handled. And so far, I just manually patch ranges in a database, but need to think how to do that automatically. (It probably would be enough to just union all ranges for all functions, ScratchABlock, the expected consumer of this stuff, will clean up unneeded basic blocks from each function).

File "scratchabit.py", line 506, in
e.loop()
"ScratchABit/pyedit/editor.py", line 254, in loop
res = self.handle_key(key)
File "scratchabit.py", line 314, in handle_key
if '0' <= e.value[0] <= '9':
IndexError: string index out of range

The automatic function detection seems to register also functions that are outside the valid areas. This causes errors when saving the file afterwards:

Traceback (most recent call last):
  File "./ScratchABit.py", line 1032, in <module>
    saveload.save_state(project_dir)
  File "/home/user/scratch/ScratchABit/scratchabit/saveload.py", line 32, in save_state
    engine.ADDRESS_SPACE.save_addr_props(project_dir + "/project.aprops")
  File "/home/user/scratch/ScratchABit/scratchabit/engine.py", line 683, in save_addr_props
    while addr > areas[area_i][END]:
IndexError: list index out of range

The following patch logs the offending function definition:

--- a/scratchabit/engine.py
+++ b/scratchabit/engine.py
@@ -680,8 +680,11 @@ class AddressSpace:
             if addr > area_end:
                 stream.close()
                 area_i += 1
-                while addr > areas[area_i][END]:
+                while area_i<len(areas) and addr > areas[area_i][END]:
                     area_i += 1
+                if area_i>=len(areas):
+                    log.debug("addr not valid (%x,%s)", addr, props.get("label"))
+                assert area_i<len(areas)

pfalcon/ida-xtensa2#1 represents an issue with the current code traversal process of SAB: it tries to follow all code paths of the current function, before considering calls the current function makes. Unfortunately, some calls are effectively jumps and never return. Trying to decode bytes after such a call may lead to conflicts in byte types, which may affect decoding of other functions following such instructions. So, instead, should try to emulate how a real CPU does - do a depth-first traversal of as many functions as possible first, and consider return addresses from calls only after that. This should allow to detect natural stop-gaps for unconstrained and incorrect code propagation. E.g. if following a call instruction there's a literal data belonging to another function, then it doesn't make sense to try to interpret this data as an instruction.

This approach will however have problems with function boundary detection (such detection was the reason to adopt breadth-first scheme). So, it will need to be reworked.

Hi Paul,

I just wanted to check if this is the recommended way to add a new plugin:
http://www.esp8266.com/viewtopic.php?f=13&t=3756&p=22201#p22200

If you let me know the recommended way, I'll submit a PR with an addition to the README about adding plugins.

Cheers,

Angus

@thesourcerer8 wrote in #27 (comment) :

I real and properly working undo function would be the greatest feature possible, I think. I guess that it's a bit difficult to pull it off in a somewhat memory efficient way due to all the side-effects every action can trigger.

Right, but memory isn't a scarce resource on modern workstation. And it would be hard to beat modern web browsers anyway (running top, I see 11 chromium processes, each with ~2GB VIRT size).

So yeah, there definitely no need to be shy to make users' life easier using the memory they have (just leave them in the control of that memory). And initial implementation can be as dumb as just making copies of all data structures (there're 2 - address space flags, one byte for each declared address space byte and address properties, more sparse, but also more weight per entry) before making an interactive change operation.

The concern here would be not memory usage, but speed of copying, I filed #37 in that regard. It should still be good for a typical under-1MB "embedded" project. And if larger projects will show issues, we can think about that later (the ideas are there - e.g. for flag space implement copy-on-write arrays, which would store only changed "pages", and props are fine-grained already, so only changes can be stored).

So, the idea is that user would be able to control levels of undo stored, and that will control amount of memory taken.

Hi tried ScratchABit now again with my test files: http://www2.futureware.at/~philipp/ssd/analyse/EXT0CB6Q.dec.P21.frmw which contain both ARM and Thumb code, and it starts with ARM code at 0x0 and has a Thumb function that starts at 0x574.
According to a comment on the Kosagi forum, ScratchABit should support both ARM and thumb now, but I couldn't get it working. This is my .def file:

#cpu any_capstone # does not work
cpu arm_32_arm_capstone # works, only ARM
#cpu arm_32_capstone # works, only ARM
#cpu arm_32_thumb_capstone # only THUMB
#cpu arm_thumb # only THUMB
show bytes 4
area .bin 0x00(0x10000) rwx
load EXT0CB6Q.dec.P21.frmw 0x0
[entrypoints]
start = 0x000

Is there a way to define where ARM and where Thumb code is in interactive mode?

PowerPC CPU is capable of running in either little-endian or big-endian modes. The required mode is usually specified in the executable itself. ELF container, for example, uses EI_DATA field of the ELF header for that purpose, see here.

The precise endianness mode must be set before disassembling.

I therefore propose to extend the CPU plugin interface with a possibility to specify the required endian mode as well as a way to retrieve endianness from container loaders.

BTW, it's not clear how IDA processor modules deal with this requirement. It looks like there is no consistent way to specify/retrieve endianness information in IDA, see this discussion.

This tickets tries to capture experience implementing new YAML-based "database" format, #3. Part of that would be migrating older format to new. But I personally don't run any "production" RE project yet, and everyone knows that open-source projects are there to scratch own's itch. Nor I have knowledge about other people using it in "production" mode (like, running long, multi-session RE projects, vs just loading up a binary and looking at a particular thing, essentially in once-off manner). With that in mind, working on migration code isn't really an exciting or fulfilling part of ScrachABit hacking, at least so far.

So, this ticket is to solicit - do you use ScrachABit in "production" mode? If so, let's discuss how to handle database migrations.

From current README:

Currently uses multiple files for "database", each storing particular type of information. Switch to a single YAML file instead?

This tickets is for discussion of this issue, opinions welcome!

Maybe it is possible, maybe not - it is up to you, just providing an idea.

Currently, there's no way to cancel text search ("/") operation in progress, which can be inconvenient.

TUI ignore terminal resizing

Occured on gnome-terminal, xterm

btw Is there "redraw screen" hotkey?

Continuing from #23 . See "orr/mov.w", "sub.w", "ldr" below.

│0802cf44 9878         ldrb     r0, [r3, #2]                                                                                           │
│0802cf46 5978         ldrb     r1, [r3, #1]                                                                                           │
│0802cf48 f268         ldr      r2, [r6, #12]                                                                                          │
│0802cf4a 41ea0021     orr/mov.w                                                                                                       │
│0802cf4e 0333         add      r3, #3                                                                                                 │
│0802cf50 0b44         add      r3, r1                                                                                                 │
│0802cf52 043a         sub      r2, #4                                                                                                 │
│0802cf54 7360         str      r3, [r6, #4]                                                                                           │
│0802cf56 f260         str      r2, [r6, #12]                                                                                          │
│0802cf58 fff732ba     b.w      0xa02c3c0                                                                                              │
│0802cf5c 5b10         asr      r3, r3, #1                                                                                             │
│0802cf5e 012b         cmp      r3, #1                                                                                                 │
│0802cf60 a5f10401     sub.w                                                                                                           │
│0802cf64 5ed0         beq      loc_0802d024                                                                                           │
│0802cf66 55f8042c     ldr                                  

I developed a tool to migrate from IDAPython capable tools to ScratchABit:
http://www.futureware.at/~philipp/ssd/ExportScratchABit.py
Copy it to the plugins directory, then you should get an menu option to export to ScratchABit in the File->ProduceFile menu. Use it generate a "export.py" file.
Then call ./ScratchABit.py yourfile.def --script export.py to run it and import the data into ScratchABit.
Feel free to ship the tool with ScratchABit.

When I click on Help-->About, SAB only says Unbound key: 'about'.

(This is of course only a very minor issue)

I've been wanting to look inside the BT/BLE controller stack at https://github.com/espressif/esp32-bt-lib, but ScratchABit doesn't currently handle SHN_COMMON sections.

Hello @pfalcon,

I tried to find the routines for just the disassembly in your code but was somewhat lost.
Could you give me a pointer? And also: How do you estimate the possibilities of transforming those into a
Micropython routine?
My idea was to have something that together with the inspect function here would allow for displaying the disassembled code of a viper/asm_thumb function?

Thanksm
Raul

While working with ScratchABit I saved my database, during the save (I think) an unhandled exception occurred and scratchabit exited. My database doesn't open any more, the stack trace on opening is:

(Snip many lines of relocation information)
<Relocation (RELA): Container({'r_info_sym': 1, 'r_offset': 7, 'r_addend': 268, 'r_info_type': 11, 'r_info': 267})>
Loading state...
Traceback (most recent call last):
  File "/home/gus/scripts/scratchabit.py", line 587, in <module>
    load_state(project_dir)
  File "/home/gus/scripts/scratchabit.py", line 539, in load_state
    engine.ADDRESS_SPACE.load_funcs(f)
  File "/home/gus/dev/ScratchABit/engine.py", line 487, in load_funcs
    start, end = l.split()
ValueError: need more than 1 value to unpack

I tried renaming the *.bak files over the newer database files, but I get the same error.

Here are the relevant files:
http://projectgus.com/misc/esp8266/xtensa_vectors.zip

Working from current git master 74323e9 and pfalcon/ida-xtensa2@cbe43ba3c3b (one commit before current master in your fork).

There's a related bug to the database corruption bug, which is that the stack trace from the unhandled exception on save was printed "inside" ncurses, so it wasn't readable (got interleaved with the UI elements). Also following exit the console mode was set wrong, needed a reset.

Sorry I don't have time to investigate this further right now.

I would like to add a web-integration, to have ScratchABit open a web-browser with a dynamic URL that gets the current position in the file, and preferrably also the filename or project name.
The goal is to enhance the static disassembler with dynamic information about actual traces, data-flows and statistics about the data and registers used. You can try it at the address 0x574

I tried a proof-of-concept by adding the following in ScratchABit.py:

elif key == b"w":
  self.show_status("Opening the current address on the Web...")
  os.system("lynx http://www2.futureware.at/cgi-bin/ssd/searchlog?q=0x"+("%08x" % (self.get_cur_line().ea))+"-\>")
  self.show_status("Done.")

It works great for me, but for an official feature, we should make the URL configurable (in the Configuration, or in the def file, which do you prefer?), and I would like your opinion on whether we should use "w" or some other hotkey for it. (I thought "w" for "web" might be a good idea.)

I have some firmware images dumped from the flash of a few IOT devices whose functionality I'm trying to re-implement using ESPHomeLib. They all appear to be in esp8266 v1 format. It appears that the esptool.py 'elf2image' command grabs various parts of the ELF binary (IROM, TEXT, etc) and packs them into the format that the on-chip bootloader wants. It does not appear to have any code to do the reverse - convert the flash image back to an ELF binary for analysis.

Is there any way I can analyze these files with ScratchABit, or is it pretty well dependent on having an ELF binary?

On Debian based systems, "apt-get install libcapstone-dev" is needed to install the capstone library before installing the python module with pip3 can work.

With 2.0, Capstone-based ARM support went online, and Capstone supports several more architectures. Fairly speaking, ARM support enablement took a bunch of effort (and isn't really complete), but the cornerstone was supporting 2nd ISA for the code in the same address space. Beyond that, Capstone seems to over pretty weak semantic characterization of instructions, so bunch of that needs to be handled in arch-specific manner in the ScratchABit plugin.

Still, it shouldn't be a rocket science to enable more archs, and this ticket is submitted in the hope to find people who'd be interested to give it a try and share feedback.

References:

• https://github.com/pfalcon/ScratchABit/blob/master/plugins/cpu/_any_capstone.py (and git log -p --follow on it)
• https://github.com/pfalcon/ScratchABit/blob/master/Makefile.examples

Hi, I'm writing a CPU plugin which calls ua_add_dref(0, xxx, dr_O) based on a value that's loaded into a register. Sometimes these values don't point into a valid address range, leading to a an error when I try to save the project:

2017-05-26 21:45:56,666 Exception processing user command
Traceback (most recent call last):
  File "/.../ScratchABit.py", line 123, in handle_input
    return super().handle_input(key)
  File "/.../ScratchABit/picotui/basewidget.py", line 69, in handle_input
    res = self.handle_key(inp)
  File "/.../ScratchABit/picotui/editor.py", line 208, in handle_key
    return self.handle_edit_key(key)
  File "/.../ScratchABit.py", line 543, in handle_edit_key
    saveload.save_state(project_dir)
  File "/.../ScratchABit/scratchabit/saveload.py", line 32, in save_state
    engine.ADDRESS_SPACE.save_addr_props(project_dir + "/project.aprops")
  File "/.../ScratchABit/scratchabit/engine.py", line 680, in save_addr_props
    fl = self.get_flags(addr)
  File "/.../ScratchABit/scratchabit/engine.py", line 247, in get_flags
    raise InvalidAddrException(addr)
scratchabit.engine.InvalidAddrException: (57259, '0xdfab')

Should I fix this in my plugin or should SAB handle this case and avoid adding the reference? What does IDA do?

[ An alternative or complement to my approach would be to create a data item with the loaded value, and let the engine add the reference automatically. AFAICS this feateure hasn't been implemented in SAB so far, though. ]

Can this program be ported to Windows using Cygwin or MinGW?

So, "mov pc, ...", "pop ..., pc" shouldn't be treated as unconditional flow-terminating instructions, flow may continue if the instruction is conditional (as usual, depends).

Saving (File->Save) seems to be slow on slow terminals, since it outputs the current location quite often. I would suggest to ouput the position less often to speed it up.

Hi, first of all, nice project!

I'd like to be able to use the mouse wheel to scroll the viewport. As far as I can see, ScratchABit would have to:

• announce its interest in mouse scroll events to the terminal
• interpret the control sequences for scroll up/down
• pass those events down into the Editor view and handle them similar to page up/down events

Pressing Ctrl+Q give me hard-to-copy popup with follwing

Traceback (most recent call last):                                  
  File "scratchabit.py", line 237, in handle_edit_key               
    return self.handle_key_unprotected(key)                         
  File "scratchabit.py", line 471, in handle_key_unprotected        
    res = d.loop()                                                  
  File "/home/user/temp/build/ScratchABit/picotui/basewidget.py", li
    self.redraw()                                                   
  File "/home/user/temp/build/ScratchABit/picotui/widgets.py", line 
    w.redraw()                                                      
  File "/home/user/temp/build/ScratchABit/picotui/editor.py", line 5
    self.show_line("", -1)                                          
  File "/home/user/temp/build/ScratchABit/picotui/widgets.py", line 
    l = self.render_line(l)[:self.width]                            
  File "scratchabit.py", line 466, in render_line                   
    return "%08x %s" % l                                            
TypeError: %x format: an integer is required, not str               

ps python3.5 example.def

It's possible to have mov [mem], offset mem. Even more it may come with analyses and transformations on top of raw machine code. For example, I already saw it with "l32r resolution" xform for xtensa.

Ignoring this fact won't work well, at least because there can be competition between different passes what value to store. The way to solve this is to record multiple values - as set, or to save memory, as string of multiple chars.