Initial support for macOS about manticore HOT 34 CLOSED

Regarding dependencies, on latest Mac OS X (Mojave) - to install capstone and unicorn one has to do:

brew install capstone && export MACOS_UNIVERSAL=no && pip install capstone
brew install unicorn && UNICORN_QEMU_FLAGS="--python=`whereis python`" pip install unicorn`

[1] re capstone: capstone-engine/capstone#1235 - the solution proposed by Inndy works fine
[2] re unicorn: unicorn-engine/unicorn#206 - somehow setting the variable as a python 3 interpreter works fine, lol

from manticore.

What should be done to verify proper execution? I've some small fixes to the tests that ensure they work on Mac OS, and I'm working through the linux examples and they seem to work just fine.

from manticore.

from manticore.

i was in the process of cleaning that stuff up, you can remove the setrlimit.

see; dev-pickle-rick

from manticore.

for all of these, /bin/ls was probably just chosen as a convenient linux ELF binary to create an initial Linux state. i don't think there was any particular requirement for it to be dynamically linked, and we have a few binaries in the repo that should do the job like defunct mentioned (tests/binaries/). i think we should prob just change all these /bin/ls references to one of these tests/binaries, as opposed to doing platform specific checks. i'd like to avoid those :)

from manticore.

The example binaries in examples/linux are configured to be statically linked, but dynamic binaries will not work on macOS. I also just added that to the comment above

from manticore.

I am using MacOS and managing to run manticore, but maybe not all the features.
The only hack I did is the pull request above.

What is left to be done here ?

from manticore.

@catenacyber check out the top comment, primarily #93 is the big one :)

from manticore.

Thanks @Mossberg : I see that I will need to improve much my pull request then

from manticore.

After closing #93, what should be next ? support Mach-O format ?

from manticore.

at this point since #93 is closed, i think the main thing is making sure all of those checkboxes in the top comment are completed, and doing testing of running manticore on mac. analyzing mach-o binaries is out of scope of this issue, we are simply looking to have mac users able to use manticore on their computers :)

from manticore.

Ok so first one

eliminate all uses of /dev (i think there are a few)

doing grep -r "/dev" manticore
I only see one : manticore/platforms/decree.py: data = file("/dev/urandom", "r").read(count)

But I have my /dev/urandom on macOS as well

Is this ok to you ?

from manticore.

ah right, macos is unixy 😅 that looks fine to me

from manticore.

Second one

make sure all python stdlib calls we make work uniformly on linux/osx

Maybe #897 is part of this, what do you think @Mossberg ?

from manticore.

yup, i'd say so

from manticore.

Any ideas on how to test all stdlib calls ?

from manticore.

hm, i'm not totally sure. i think it might just be good enough to do a cursory review of the codebase and document any stdlib calls we make that are particularly suspect for non-portability.

from manticore.

Hmm I googled "site:https://docs.python.org/3/library/ macos" and looked out in python documentation which calls would be concerned.
I found functions such as os.getgroups but none used in manticore...
So, I guess that we can tick the second box.
How we can tick the third one (dependencies) ? trying a fresh install is enough ?

from manticore.

Hm, i'm thinking that if our goal is to have beta/experimental support for macos it doesn't have to work absolutely perfectly. If tests pass on macos maybe that would be enough to announce beta macos support and resolve this issue. then we can fix individual macos bugs as they are reported.

from manticore.

Hmm...

manticore/utils/event.py", line 38
    class Eventful(object, metaclass=EventsGatherMetaclass):
                                    ^
SyntaxError: invalid syntax

I will look into it

from manticore.

Ok, now I am getting

manticore/core/workspace.py", line 52, in serialize
    resource.setrlimit(resource.RLIMIT_STACK, [0x100 * maxlim, resource.RLIM_INFINITY])
ValueError: not allowed to raise maximum limit

from manticore.

Thanks @defunctio I was trying to keep it, but make it work with reasonable values on Mac

from manticore.

Next error is

manticore.ethereum.EthereumError: Solidity compiler not installed.

Maybe this can be added to the wiki
https://github.com/trailofbits/manticore/wiki/Hacking-on-Manticore

from manticore.

I was thinking not too long ago maybe we should consider using py-solc instead of our own wrapper. This does come with an added benefit of solc version management for both linux and osx

from manticore.

maybe we should consider using py-solc instead of our own wrapper.

indeed but I think that deserves another issue

As for Mac, here is the next error :

======================================================================
ERROR: testCreating (tests.test_driver.ManticoreDriverTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "manticore/tests/test_driver.py", line 27, in testCreating
    m = Manticore('/bin/ls')
  File "manticore/manticore/manticore.py", line 188, in __init__
    self._initial_state = make_initial_state(path_or_state, argv=argv, **kwargs)
  File "manticore/manticore/manticore.py", line 138, in make_initial_state
    raise NotImplementedError("Binary {} not supported.".format(binary_path))
NotImplementedError: Binary /bin/ls not supported.

Indeed, my /bin/ls is a Mach-O file
Should a linux /bin/ls binary be added to the manticore tests ?

We come close to the end
(+ one path patch for the tests)

from manticore.

Unless that was intended to test dynamically linked binaries (which I don't think it is?) you could probably just switch that to use tests/binaries/basic_linux_amd64 or something

from manticore.

Unless that was intended to test dynamically linked binaries (which I don't think it is?)

This comment shows that this may be the case :

'''
    TODO(mark): these tests assumes /bin/ls is a dynamic x64 binary
    '''

from manticore.

I would probably just do some platform checks for now and if the test environment is OSX and skip tests that rely on dynamically linked ELF targets and leave a #TODO/#FIXME behind. The CI will end up testing for it anyways.

Anyone else may have a more elegant solution?

from manticore.

@Mossberg you wrote the above comment (these tests assumes /bin/ls is a dynamic x64 binary)

Do you have a better proposal than @defunctio ?

PS : the following tests use /bin/ls, even if only test_linux.py has the comment :

• tests/test_driver.py
• tests/test_linux.py
• tests/test_models.py
• tests/test_state.py
• tests/test_unicorn.py
• tests/test_workspace.py

from manticore.

And now it works with #1032 :-)

Ran 5407 tests in 459.774s

OK (SKIP=5)

from manticore.

@catenacyber a question about your environment with those tests running; are you on python 3.6? and also was unicorn installed? i am having issues getting tests to pass, and it's mostly because unicorn does not seem to install in my macOS python 3.6 environment

specifically, i hit this error when installing unicorn

https://github.com/unicorn-engine/unicorn/blob/0109cd6c8a2b268d0aaf9e597752682508c7ffed/qemu/configure#L555-L561

i can install unicorn in a Linux python3.6 environment because it seems there is a wheel for it, so there is no need to compile.

from manticore.

I am running Python 3.7.0
And I seem to have unicorn module installed

> pip freeze | grep unicorn
unicorn==1.0.1

I do not remember installing it though

from manticore.

I added the description to https://github.com/trailofbits/manticore#installation , we probably can't enforce this on setup.py so I assume this is all we can do here.

from manticore.

so I assume this is all we can do here

Could we make that unicorn supports python3 ? (without needing the trick)

from manticore.