Code Monkey home page Code Monkey logo

exodus-standalone's Introduction

εxodus standalone

Build Status

εxodus CLI client for local APK static analysis.

Summary

Using Docker

The easiest way to analyze an APK is to use our Docker image.

Simply go to the directory where the APK file is and run:

docker run -v $(pwd):/app --rm -i exodusprivacy/exodus-standalone /app/<your apk file>

Manual usage

Installation

Clone this repository:

git clone https://github.com/Exodus-Privacy/exodus-standalone.git
cd exodus-standalone

Install dexdump:

sudo apt-get install dexdump

Create Python virtualenv:

sudo apt-get install virtualenv
virtualenv venv -p python3
source venv/bin/activate

Download and install dependencies:

pip install -r requirements.txt

Analyze an APK file

Usage

$ ./exodus_analyze.py --help
usage: exodus_analyze.py [-h] [-t] [-j] [-o OUTPUT_FILE] [-i IGNORE] [-e CODE] apk

positional arguments:
  apk                   the apk file to analyse

optional arguments:
  -h, --help            show this help message and exit
  -t, --text            print textual report (default)
  -j, --json            print JSON report
  -o OUTPUT_FILE, --output OUTPUT_FILE
                        store JSON report in file (requires -j option)
  -i IGNORE, --ignore IGNORE
                        comma-separated ids of trackers to ignore
  -e CODE, --exit-code CODE
                        use the CODE instead of trackers counter as exit code if trackers was detected

Text output

./exodus_analyze.py my_apk.apk

be sure to activate the Python virtualenv before running exodus_analyze.py.

Example:

=== Informations
- APK path: /tmp/tmp1gzosyt4/com.semitan.tan.apk
- APK sum: 8e85737be6911ea817b3b9f6a80290b85befe24ff5f57dc38996874dfde13ba7
- App version: 5.7.0
- App version code: 39
- App name: Tan Network
- App package: com.semitan.tan
- App permissions: 9
    - android.permission.INTERNET
    - android.permission.ACCESS_NETWORK_STATE
    - android.permission.ACCESS_FINE_LOCATION
    - android.permission.WRITE_EXTERNAL_STORAGE
    - android.permission.READ_PHONE_STATE
    - android.permission.VIBRATE
    - com.semitan.tan.permission.C2D_MESSAGE
    - com.google.android.c2dm.permission.RECEIVE
    - android.permission.WAKE_LOCK
- App libraries: 0
=== Found trackers
 - Google Analytics
 - Google Ads
 - Google DoubleClick

JSON output

./exodus_analyze.py -j [-o report.json] my_apk.apk

be sure to activate the Python virtualenv before running exodus_analyze.py.

Example:

{
  "trackers": [
    {
      "id": 70,
      "name": "Facebook Share"
    },
    [...]
  ],
  "apk": {
    "path": "com.johnson.nett.apk",
    "checksum": "70b6f0d9df432c66351a587df7b65bea160de59e791be420f0e68b2fc435429f"
  },
  "application": {
    "version_code": "15",
    "name": "Nett",
    "permissions": [
      "android.permission.INTERNET",
      "android.permission.ACCESS_NETWORK_STATE",
      "android.permission.WRITE_EXTERNAL_STORAGE",
      "android.permission.READ_PHONE_STATE",
      "android.permission.READ_EXTERNAL_STORAGE",
      "android.permission.WAKE_LOCK",
      "com.google.android.c2dm.permission.RECEIVE",
      "com.johnson.nett.permission.C2D_MESSAGE"
    ],
    "version_name": "1.1.12",
    "libraries": [],
    "handle": "com.johnson.nett"
  }
}

Pitfalls

This tool uses dexdump and only provides GNU/Linux x86_64 version of it.

Download an APK from an εxodus instance

Configuration

Create config.py file in the project directory specifying:

CONFIG = {
    'username': 'alice',
    'password': 'bob',
    'host': 'http://localhost:8000'
}

Usage

$ ./exodus_download.py --help
usage: exodus_download.py [-h] report_id destination

positional arguments:
  report_id    the report of the app to download
  destination  the destination folder

optional arguments:
  -h, --help   show this help message and exit

be sure to activate the Python virtualenv before running exodus_download.py.

Example of output

./exodus_download.py 15 /tmp/
Successfully logged in
Downloading the APK ...
APK successfully downloaded: /tmp/fr.meteo.apk

Continuous Integration

You can use εxodus-standalone in your CI pipelines.

Below are listed some examples of how to integrate it.

GitLab CI/CD

exodus_scan:
  stage: audit
  image:
    name: exodusprivacy/exodus-standalone:latest
    entrypoint: [""]
  script:
    - /exodus_analyze.py [YOUR_APK_PATH]

GitHub Actions

steps:
  - name: Execute exodus-standalone
    uses: docker://exodusprivacy/exodus-standalone:latest
    with:
      args: /github/workspace/[YOUR_APK_PATH]

exodus-standalone's People

Contributors

dmyachin avatar gu1nness avatar jean-baptistec avatar milouse avatar pnu-s avatar smandon avatar u039b avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

exodus-standalone's Issues

installation instructions omit steps and information

It's unclear from the installation instructions which commands need to run as root and which run as a user. There also seems to be an underlying assumption that users are familiar with python usage. My guesswork made for a rough experience but it worked out in the end. Here's a walk-through:

Looking ahead, a package is needed (dexdump). So that should be installed first because if that fails there's no point in doing the manual steps.

sudo aptitude install dexdump

It's clear that I must choose a location and it's likely that root should run this since I'm doing a systemwide install. So root does cd /usr/local/src/, followed by:

$ torsocks git clone https://github.com/Exodus-Privacy/exodus-standalone.git
$ cd exodus-standalone

As a user:

nano ~/.config/gplaycli/gplaycli.conf

Then as root:

$ virtualenv venv -p python3
bash: virtualenv: command not found

Oops, missed a package. That should be added to the first step.

$ sudo aptitude install virtualenv
$ virtualenv venv -p python3
$ source venv/bin/activate

This next step is labeled "Install dependencies":

$ pip install -r requirements.txt

I don't think I've used pip before, but I wish I had realized that it would download stuff from the WAN, so that I would have known to prefix torsocks. The instructions should really say "download and install dependencies", to prompt Tor users to make arrangements. I was expecting the git clone to have done the downloading.. I wasn't careful enough to notice how little came from the clone and to then realize that pip would download stuff. Others will likely get stung by that too.

So now that installation is complete, as a user I run:

$ python /usr/local/src/exodus-standalone/exodus_analyze.py -h
Traceback (most recent call last):
  File "/usr/local/src/exodus-standalone/exodus_analyze.py", line 5, in <module>
    from exodus_core.analysis.static_analysis import StaticAnalysis
ImportError: No module named exodus_core.analysis.static_analysis

If root runs that command inside the virtualenv then it works, but root only happened to be in the virtualenv as part of the installation process, which is now over. When root does a control-d to exit that virtualenv, the whole shell is killed off including the parent. That's also astonishing. So something apparently did an exec to avoid forking. Whatever the proper way to exit that environment is, it should be documented.

So I first figured the virtualenv command needs to run every time. But that errors. After doing source venv/bin/activate as a user, it worked. So the activate script should be repeated in the "Analyze an APK file" steps. So this is how the instructions should say to run the tool:

$ source <root pkg dir>/exodus-standalone/venv/bin/activate
$ python <root pkg dir>/exodus-standalone/exodus_analyze.py "$apkfile"

Note that some APK files cause that to barf up this:

Traceback (most recent call last):
  File "/usr/local/src/exodus-standalone/exodus_analyze.py", line 56, in <module>
    analysis = AnalysisHelper(apk_file)
  File "/usr/local/src/exodus-standalone/venv/lib/python3.5/site-packages/exodus_core/analysis/static_analysis.py", line 96, in __init__
    self.load_apk()
  File "/usr/local/src/exodus-standalone/venv/lib/python3.5/site-packages/exodus_core/analysis/static_analysis.py", line 130, in load_apk
    self.apk = APK(self.apk_path)
  File "/usr/local/src/exodus-standalone/venv/lib/python3.5/site-packages/androguard/core/bytecodes/apk.py", line 117, in __init__
    self.zip = zipfile.ZipFile(io.BytesIO(self.__raw), mode="r")
  File "/usr/lib/python3.5/zipfile.py", line 1026, in __init__
    self._RealGetContents()
  File "/usr/lib/python3.5/zipfile.py", line 1094, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

But that's related to the APK.. some APKs work.

Anyway, I think I'm sorted now. But the instructions need to guide people away from the above pitfalls. It would also be good to state whether the activate or exodus_analyze.py scripts need the Internet, so users can firejail and or torsocks it. A firejail profile would also perhaps be useful.

Provide a way to ignore trackers and avoid the script to fail

There should be a way, for instance with a file called .exodusignore to ignore some trackers so that exodus-standalone can be used in cases where some trackers are expected.

We need to think how to identify trackers (name is not necessarily stable, id is probably a bit more)

Running under MacOS

Hi,

I tried following install instructions on my Mac and got

$ pip install -r requirements.txt
Collecting https://github.com/Exodus-Privacy/exodus-core/releases/download/v1.0.4/exodus_core-1.0.4.tar.gz (from -r requirements.txt (line 1))
  Downloading https://github.com/Exodus-Privacy/exodus-core/releases/download/v1.0.4/exodus_core-1.0.4.tar.gz (771kB)
    100% |████████████████████████████████| 778kB 717kB/s
    Complete output from command python setup.py egg_info:
    Unfortunately, we do not support your platform darwin

    ----------------------------------------

I guess exodus-standalone only works on Linux? Not sure how difficult it would be to support mac as well? If that's a big deal, what about offering a Docker image or something like this for easy local APK analysis on any system?

Thanks for the work you put in this software!

Migrate to python 3.11

We have planned to migrate exodus to debian 12 and python 3.11.
It's can be good also to update exodus-standalone.
Files need to be updated:

  • Workflow Github
  • DockerFile

'str' object has no attribute 'decode' with --json option

Hello,

I am getting the reverse error of #5:

Traceback (most recent call last):
  File "exodus_analyze.py", line 62, in <module>
    report = json.dumps(analysis.create_json_report(), indent = 2)
  File "exodus_analyze.py", line 18, in create_json_report
    'libraries': [l.decode('utf-8') for l in self.get_libraries()],
  File "exodus_analyze.py", line 18, in <listcomp>
    'libraries': [l.decode('utf-8') for l in self.get_libraries()],
AttributeError: 'str' object has no attribute 'decode'

get_libraries() already returns strings so there is no need to call decode().

No module named 'exodus_core' when trying to run

Hello, I am trying to install / run exodus standalone, but with no luck on centos7/local vm.

i have done steps ( per readme)
x) git clone https://github.com/Exodus-Privacy/exodus-standalone.git
x) cd exodus-standalone
x) virtualenv venv -p python3
x) source venv/bin/activate
x) pip install -r requirements.txt

some of it failed so had to do (from some basic troubleshooting/searchengine)
x) pip install ez_setup
x) pip install --upgrade setuptools
x) pip install --upgrade setuptools
and also
x) pip install --upgrade -r requirements.txt

And then it said install successful but when trying to run it it cannot find exodus-core.

(venv) [exodus@localhost exodus-standalone]# ./venv/bin/python exodus_analyze.py test.apk
Traceback (most recent call last):
File "exodus_analyze.py", line 4, in
from exodus_core.analysis.static_analysis import StaticAnalysis
ImportError: No module named 'exodus_core'
(venv) [exodus@localhost exodus-standalone]#

Any help / idea ?
Should i go for another Linux flavor ?
thanks.

[Feature request] Web UI

Hello,

It would be nice to have a GUI allowing to upload an APK in browser and get a graphical report.

Thanks

Errors in terminal with latest version

1.4.0 version generates errors in terminal (detected in exodus android app Github actions) :

dexdump W 12-26 08:30:24     7     7 dex_file_verifier.cc:3172] This dex file is invalid and will be rejected in the future. Error is: Interface field is not public final static, Lcom/android/tools/r8/androidapi/h;.$desugar$clinit: 1008(static )
dexdump W 12-26 08:30:26    13    13 dex_file_verifier.cc:3172] This dex file is invalid and will be rejected in the future. Error is: Interface field is not public final static, Lcom/android/tools/r8/utils/structural/k;.$desugar$clinit: 1008(static )
dexdump W 12-26 08:30:26    13    13 dex_file_verifier.cc:3172] This dex file is invalid and will be rejected in the future. Error is: Interface field is not public final static, Lcom/android/tools/r8/utils/structural/s;.$desugar$clinit: 1008(static )
dexdump W 12-26 08:30:26    13    13 dex_file_verifier.cc:3172] This dex file is invalid and will be rejected in the future. Error is: Interface field is not public final static, Lcom/android/tools/r8/internal/oo;.$desugar$clinit: 1008(static )

No matching manifest for linux/arm64/v8 in the manifest list entries

Description

When running docker pull exodusprivacy/exodus-standalone command I'm getting following error

Using default tag: latest
latest: Pulling from exodusprivacy/exodus-standalone
no matching manifest for linux/arm64/v8 in the manifest list entries

image

Platform Info

MacOS Apple M2 Pro

Add contributing documentation

We should add a CONTRIBUTING.md file to make contributing easier.

Things we should mention in it:

  • the flake 8 linter
  • the tests (how to add new ones, how to run them)

Stable release

Hello,
Are stable release tags planned for this project? It would make it easier to know when to update.

`exodus_analyze.py` crash analyzing a `aab` format application

Here is the log:

python3.8 exodus_analyze.py ~/StudioProjects/Pilldroid/app/product/release/app-product-release.aab 
Missing AndroidManifest.xml. Is this an APK file?
=== Information
- APK path: /home/jacques/StudioProjects/Pilldroid/app/product/release/app-product-release.aab
- APK sum: f978a7ce751609a750341d099fd498f0953865df949d2263548e5848b152e744
Traceback (most recent call last):
  File "exodus_analyze.py", line 78, in <module>
    main()
  File "exodus_analyze.py", line 71, in main
    analysis.print_apk_infos()
  File "/home/jacques/exodus-standalone/test_app_venv/lib/python3.8/site-packages/exodus_core/analysis/static_analysis.py", line 471, in print_apk_infos
    print('- App version: {}'.format(self.get_version()))
  File "/home/jacques/exodus-standalone/test_app_venv/lib/python3.8/site-packages/exodus_core/analysis/static_analysis.py", line 239, in get_version
    return self.apk.get_androidversion_name()
  File "/home/jacques/exodus-standalone/test_app_venv/lib/python3.8/site-packages/androguard/core/bytecodes/apk.py", line 620, in get_androidversion_name
    return self.androidversion["Name"]
KeyError: 'Name'

With unzip -l ~/StudioProjects/Pilldroid/app/product/release/app-product-release.aab I actually can't find the AndroidManifest.xml file.

Split apk support?

How can we analyse them?

It does not seem to work when I just analyse one or all APKs I can opull from Android.

Edit: Ok, usually analysing the base.apk may be enough.

Detect dexdump crash

I install the standalone for testing some apks but trackers list is always empty.
For example

=== Information
- APK path: fr.playsoft.lefigarov3.apk
- APK sum: 51532dd165e38e2d0ec11dc119b152bcd7808d9feb09f15c7ed4542987776e67
- App version: 5.1.3
- App version code: 94
- App UID: FABCC978B5CEFF042A9A462A3922ECA956B2420B
- App name: Le Figaro
- App package: fr.playsoft.lefigarov3
- App permissions: 10
    - android.permission.INTERNET
    - android.permission.ACCESS_NETWORK_STATE
    - android.permission.WRITE_EXTERNAL_STORAGE
    - android.permission.RECEIVE_BOOT_COMPLETED
    - android.permission.VIBRATE
    - com.google.android.c2dm.permission.RECEIVE
    - com.android.vending.BILLING
    - android.permission.WAKE_LOCK
    - fr.playsoft.lefigarov3.permission.C2D_MESSAGE
    - android.permission.ACCESS_WIFI_STATE
- App libraries: 0
- Certificates: 1
    - Issuer: countryName=FR, stateOrProvinceName=Ile de France, localityName=Paris, organizationName=Le Figaro, organizationalUnitName=LeFigaro Nouveaux médias, commonName=Francois Rannou 
Subject: countryName=FR, stateOrProvinceName=Ile de France, localityName=Paris, organizationName=Le Figaro, organizationalUnitName=LeFigaro Nouveaux médias, commonName=Francois Rannou 
Fingerprint: e9cdbbbd1d6a307563368c572e2efe2e106dbf4f 
Serial: 1317228289
=== Found trackers: 0

Is not exactly the same version on https://reports.exodus-privacy.eu.org/reports/2296/
But all 17 trackers seems disappeared ?
I have tested this behavior with 3 different apk from playstore and all of them have zero trackers.

I have install the exodus-core with pip in a venv version 1.0.8.
The md5sum of fr.playsoft.lefigarov3.apk is 7bc323c6ac8a146cff8753175b3ec60d , I can supply it if needed for some test.

My install have certainly something wrong ?

Add --exit-code option

Add an option exit-code to allow the script user to change the returned exit code in case of trackers found.

Could be used for instance to run with --exit-code 0 to avoid failing or --exit-code 1 to avoid the current exit codes (based on the number of found trackers).

Maximum API level reached

I do analyze an app, and although everything seems to work, it returns me an error like this:

Requested API level 29 is larger than maximum we have, returning API level 28 instead.

Unfortunately that "error" (is it even an error or only a warning/note etc.?) is not really descriptive and I don't understand what this means now.

I.e. what implications does this have?

The analysis looks fine…

Add cache feature for εxodus trackers database

Currently, exodus-standalone downloads εxodus trackers database for each run.

The database being fairly stable (changing only from time to time), the script should store it locally so that it is not downloaded every time (less load on εxodus servers, faster execution, etc.)

What the script could do :

  • if no data locally, download it and store it
  • if some data is available, check its timestamp and only refresh it if older than X
  • if an option such as --refresh-cache is provided, refresh the case anyway

Reorganize repository structure

Feels like the current rule of "everything in the root directory" is not ideal

We could reorganize the repository to follow python best practices (at least separate tests from the rest)

JUnit XML output format

Add JUnit XML output format so that exodus-standalone can be integrated more easily in some CI systems

Feature: Upload report to Exodus website?

I just spent quite a while trying to get a privacy report that couldn't be generated by the exodus privacy website (the site couldn't download the APK for some reason).

I ended up manually running some commands on a docker container running in Azure, and got the report. Cool!

However, now I'm the only person that knows the trackers/permissions requested by this app. It seems like there should be an optional mechanism for me to upload the report to the master database, so other users can capitalize on my findings. Maybe a -u flag as part of the analyzer python script?

TypeError: object of type 'generator' has no len()

I installed exodus-standalone within venv and get

`ile "exodus_analyze.py", line 59, in
analysis.print_apk_infos()
File "exodus-standalone/venv/lib/python3.7/site-packages/exodus_core/analysis/static_analysis.py", line 483, in print_apk_infos
print('- App libraries: %s' % len(libraries))
TypeError: object of type 'generator' has no len()``


Commenting the three lines makes it run again, but then of course no libraries are shown ;-)

Add the possibility to use a .exodusignore file

Currently we only can ignore trackers with the --ignore option which is fairly basic and uses tracker id.

Using a separate file could make this more clear, and especially allow to:

  • add comments
  • ignore trackers by name (although that's a bit risky as names can change)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.