Code Monkey home page Code Monkey logo

ida_ipython's Introduction

What's New

  • Improve python plugin load order (prevents crashes when python does not load correctly)
  • Add in safe IDA process termination (Special thanks to @tmr232 for this)

What and Why?

This is a plugin to embed an IPython kernel in IDA Pro. The Python ecosystem has amazing libraries (and communities) for scientific computing. IPython itself is great for exploratory data analysis. Using tools such as the IPython notebook make it easy to share code and explanations with rich media. IPython makes using IDAPython and interacting with IDA programmatically really fun and easy.

Example Uses

QT Console

You can just use IPython qtconsole for a better interactive python shell for IDA.

Image of Basic QT Usage

You can also use the QT console to graph things. This is an example creating a bar chart for the occurrences of each instruction mnemonic in a function (in notepad.exe).

Image of QT with graph

Notebooks

Another useful case is using IPython notebooks.

  • Function Entropy - Here is an example where we compute the entropy (using scipy stats module) of each function in notepad.exe and graph the result.
  • Cython and IDA - Here is an example where we use the cython cell magic to call IDA Api's that are not exposed via IDAPython.
  • Sark Snapshots - Example of screen snapshots using Sark.

More examples..soon...

How the plugin works

IDA is predominantly single threaded application, so we cannot safely run the kernel in a separate thread. So instead of using another thread a hook is created on the QT process events function and the do_one_iteration method of the ipython kernel is executed each frame.

Installation

I suggest using the Anaconda distribution of Python as it comes with all the required python libraries pre-built and installed. To get IDA to use Anaconda, simply set the PYTHONHOME enviroment variable. Alternatively you can install IPython and the dependencies separately.

This plugin should work on all 6.X x86 QT versions of IDA on Windows.

Basic Installation and QTConsole

  1. Download and extract the release
  2. Copy the contents of the plugins and python directories under IDA's installation directory.
  3. Launch IDA.
  4. At the command line (Windows), start an IPython qtconsole with the kernel instance (outputted in the IDA console) e.g ipython qtconsole --existing kernel-4264.json

Using the Notebook

  1. Copy idc directory to your IDA directory. (the nothing.idc script is used to pass command line parameters to the plugin)
  2. Change the paths to the idaq.exe and idaq64.exe executables in the kernel.json under the notebook\kernels\ida32 and notebook\kernels\ida64 directories respectively
  3. Install the kernels using jupyter-kernelspec install (e.g. jupyter-kernelspec install --user notebook\kernels\ida64)
  4. When starting a notebook, choose the IDA32 or IDA64 kernels, depending on your desired IDA version.

How to Build

  1. Install cmake
  2. At the command line cd to the root directory and run the following
  3. mkdir build
  4. cd build
  5. cmake -G "Visual Studio 11" -DPYTHON_DIR="<YOUR_PYTHON_DIR>" -DIDA_SDK="<YOUR_IDASDK_LOCATION>" -DIDA_DIR="<YOUR_IDA_DIRECTORY>" .. e.g. cmake -G "Visual Studio 11" -DPYTHON_DIR="C:\Anaconda" -DIDA_SDK="C:\dev\IDA\idasdks\idasdk64" -DIDA_DIR="C:/Program Files (x86)/IDA 6.4" ..
  6. cmake --build . --config Release

So far only tested with "Visual Studio 11" compiler.

Changelog

0.5

  • Improve python plugin load order (prevents crashes when python does not load correctly)
  • Add in safe IDA process termination (Special thanks to @tmr232 for this)

0.4

  • Added IDA 6.9 support

0.3

  • 2015-10-04: Running the plugin from the plugins menu or view menu will now launch an IPython QTConsole. Consoles are terminated on exit.
  • 2015-10-04: Only capture standard output and error when executing the kernel
  • 2015-10-21: Improve error reporting

0.2

  • 2015-09-30: Added support for Jupyter (replaces original support for IPython).
  • 2015-10-01: Added support for ida64.

0.1

  • First release

To do/Future Ideas

  • More examples
  • Create a library for cell/line magic functions specific to IDA

ida_ipython's People

Contributors

james91b avatar obilodeau avatar tmr232 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ida_ipython's Issues

Bad Release 0.3

Seems like you uploaded an old file as the release ZIP. It lacks the termination of QtConsoles from the current master.

Issues with Exceptions

This
image

Happens whenever I raise an exception in the IPython shell. IDA's own shell just prints the exception to the IPython window (weird too).

I'll try and look into it. The same thing happened to me with my pure-Python implementation, so for a long while I was sure my new code is the issue, but I seem to get it with current release code as well.

Enable multiple notebook kernels

While IPython allowed to easily change profiles and kernels with --profile and other options, Jupyter does not.

The solution of using the JUPYTER_CONFIG_DIR environment variable works, but it is a very bad idea. Luckily, there is an alternative.

Jupyter allows for additional kernels to be added to the notebook using simple JSON files (docs here, examples here). The kernel.json files should be placed into a kernels/<kernel-name> directory under one of the data directories:

>>>jupyter --paths
config:
    C:\Users\user\.jupyter
    c:\python27\etc\jupyter
    C:\ProgramData\jupyter
data:
    C:\Users\user\AppData\Roaming\jupyter
    c:\python27\share\jupyter
    C:\ProgramData\jupyter
runtime:
    C:\Users\user\AppData\Roaming\jupyter\runtime

I tried using the following JSON:

{
 "display_name": "IDA64", 
 "language": "python", 
 "argv": [
  "python.exe", 
  "-c", "from idakernelmanager import main; main()",
  "-f", "{connection_file}"
 ], 
 "codemirror_mode": {
  "version": 2, 
  "name": "ipython"
 }
}

With the following code for main:

def main():
    kernel_cmd = [IDA_EXE] + sys.argv[-2:]
    return launch_ida_kernel(kernel_cmd)

It shows up nicely in the notebook,

default

But trying to launch the IDA64 kernel always yields the same error:

default

Any ideas?

Terminate IPython on IDA termination

I think the IPython console should terminate when IDA terminates. Or at least get some sort of notification.

The issue I keep running into is that I open multiple instances of IDA, and multiple instances of the qtconsole. All the consoles have the same title, and there is no indication when I close IDA. So I have multiple shells, and I have to actually type in code to see which one are still active ๐Ÿ˜ญ

The possible solutions I can think of are:

  1. If IDA spawns the qtconsole, it should also terminate it (easy, just save process information and kill it). This is kinda safe, as you can't really attach the console to multiple instances.
  2. Somehow indicate to Jupyter that the kernel has died.
  3. Set the title of the qtconsole window to match the IDA window.

The first seams the easiest.

IDAPython crashes when IDASkins shows "First Start" dialog.

This is a clone of the issue I opened in IDASkins - zyantific/IDASkins#9

I have IDAIPython installed for quite a long time, and it is working well.
Today I decided to try IDASkins. After installing it, IDA kept crashing on startup when starting to show the "First Start" dialog.
The crash seems to be related to the Python interpreter state, so I disabled my only non-default Python plugin - IDAIPython. Once done, everything worked as expected.
After the first run, I enabled IDAIPython, and all seems to work well.

I did not have time to dig in and see if this is a bug in IDAIPython or IDASkins, so I am opening an issue there as well, and referencing this one.

Use local namespace

Currently, the IPython console and the regular Python console inside IDA have different namespaces.

Is it possible to merge those namespaces?

ImportError: DLL load failed: %1 is not a valid Win32 application.

I am getting this exception. It's on a Windows 10 64bit Machine. Anaconda 32bit is installed. I made sure that any other python installations were removed.

The funny thing is that there is no error when I import from ipykernel.kernelapp import IPKernelApp in the python shell or Jupyter QtConsole.

Here is the full stacktrace:

Traceback (most recent call last):
  File "C:\devtools\IDA 6.8\python\ipythonEmbed.py", line 19, in <module>
    from ipykernel.kernelapp import IPKernelApp
  File "C:\Program Files (x86)\Anaconda2\lib\site-packages\ipykernel\__init__.py", line 2, in <module>
    from .connect import *
  File "C:\Program Files (x86)\Anaconda2\lib\site-packages\ipykernel\connect.py", line 13, in <module>
    from IPython.core.profiledir import ProfileDir
  File "C:\Program Files (x86)\Anaconda2\lib\site-packages\IPython\__init__.py", line 48, in <module>
    from .core.application import Application
  File "C:\Program Files (x86)\Anaconda2\lib\site-packages\IPython\core\application.py", line 25, in <module>
    from IPython.core import release, crashhandler
  File "C:\Program Files (x86)\Anaconda2\lib\site-packages\IPython\core\crashhandler.py", line 28, in <module>
    from IPython.core import ultratb
  File "C:\Program Files (x86)\Anaconda2\lib\site-packages\IPython\core\ultratb.py", line 124, in <module>
    from IPython.utils import path as util_path
  File "C:\Program Files (x86)\Anaconda2\lib\site-packages\IPython\utils\path.py", line 19, in <module>
    from IPython.utils.process import system
  File "C:\Program Files (x86)\Anaconda2\lib\site-packages\IPython\utils\process.py", line 15, in <module>
    from ._process_win32 import system, getoutput, arg_split, check_pid
  File "C:\Program Files (x86)\Anaconda2\lib\site-packages\IPython\utils\_process_win32.py", line 21, in <module>
    import ctypes
  File "C:\Program Files (x86)\Anaconda2\Lib\ctypes\__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed: %1 is not a valid Win32 application.
A Python Error Occurred trying to start the kernel!

Error: plugins\ida_ipython.plw cant load file

Hi have some trouble to install release 0.3/0.4.

Infos:
OS = Windows 7 64 VM
IDAPro 6.6 Portable
python 2.7.11 Anaconda 2.4.1 installed to C:\Anaconda2
set PYTHONHOME=C:\Anaconda2 global for all user
copy python/ipythonEmbed.py to C:\Users\user\Desktop\IDAPro6.6\python\ipythonEmbed.py
copy plugins/ida_* to C:\Users\user\Desktop\IDAPro6.6\plugins\

execute C:\Users\user\Desktop\IDAPro6.6\idaq.exe

ida pro console:

LoadLibrary(C:\Users\user\Desktop\IDAPro6.6\plugins\ida_ipython.plw) error: Das angegebene Modul wurde nicht gefunden.
C:\Users\user\Desktop\IDAPro6.6\plugins\ida_ipython.plw: can't load file

Translate: Das angegebene Modul wurde nicht gefunden. => The module can not be found

Do I miss something? The file C:\Users\user\Desktop\IDAPro6.6\plugins\ida_ipython.plw exist and the user has rwx permissions.

IDAIpython crash on IDA 6.9 Service Pack 1 - Windows 7

Hello,

Was working in 6.8 but after upgrading in 6.9 then 6.9 SP1 got this error:

Traceback (most recent call last):
File "C:\Program Files (x86)\IDA 6.9\python\ipythonEmbed.py", line 19, in
from ipykernel.kernelapp import IPKernelApp
ImportError: No module named ipykernel.kernelapp
A Python Error Occurred trying to start the kernel!
alleycat.py: Traceback (most recent call last):
File "C:\Program Files (x86)\IDA 6.9\python\idaapi.py", line 589, in IDAPython_ExecScript
scriptpath = os.path.dirname(script)
File "C:\Anaconda\Lib\ntpath.py", line 177, in dirname
return split(p)[0]
File "C:\Anaconda\Lib\ntpath.py", line 150, in split
while head2 and head2[-1] in '/':
AttributeError: 'module' object has no attribute 'start'

Can you fix it ? Thanks a lot.
Love your plugins, can't work anymore without it.

Linux and OS X support

It would be nice if this plugin would be cross-platform.

Personally I would like to see the Qtconsole work ported first and to Linux first of course ;)

I'm not great with win32, C or C++ but I might be able to help with small things or test.

I'll send you a PR to state that this is Windows-only since I've spent a bit of time doing my python 2.7 32-bit anaconda install in order to install your plugin only to realize later that the archive contained PE files and not ELFs... Relevant documentation ending with the dead-end ;) https://gist.github.com/obilodeau/9006a01ba8b1cd0d7350

Seems like nice work! Thanks for sharing it.

IDAIPython crashes IDA if IDAPython is not loaded.

For some reason, installing Python 2.7.11 prevented IDAPython from working on my PC. As a result - IDAIPython kept crashing IDA, resulting in a dump file.

I am not yet sure why this happens, but we should probably check for IDAPython being loaded anyway.

Adding IPython Magics

Trying to add IPython magics I came to a problem - it seems that I have to add them from the IPython console itself. Is there any other way to do it in the context of the plugin?

Being able to add magics from other plugins will be quite powerful.

IDA crashing on termination.

It seems that every time I try and close IDA, it crashes.

It is not critical, as it crashes after saving the database, but it is still a bug. As it does no noticeable harm, I only noticed it after setting Windbg to be the postmortem debugger. But once I set it, I now see IDA crashing due to an access violation on every termination.

This has been reproduced on multiple computers.

Will add more information as soon as my debugging yields any.

Could not work with IDA6.8

Traceback (most recent call last):
File "C:\Program Files (x86)\IDA 6.8\python\ipythonEmbed.py", line 19, in
from ipykernel.kernelapp import IPKernelApp
File "C:\Program Files (x86)\IDA 6.8\python\ipykernel_init_.py", line 2, in
from .connect import *
File "C:\Program Files (x86)\IDA 6.8\python\ipykernel\connect.py", line 14, in
from IPython.paths import get_ipython_dir
ImportError: No module named paths
A Python Error Occurred trying to start the kernel!
Totalcmd64_T0kbgNRUUv

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.