ninoseki / eml_analyzer Goto Github PK

View Code? Open in Web Editor NEW

229.0 9.0 40.0 11.98 MB

An application to analyze the EML file

Home Page: https://eml-analyzer.herokuapp.com/

License: MIT License

Dockerfile 1.35% Python 57.86% JavaScript 0.33% HTML 0.88% Vue 19.98% TypeScript 19.61%

eml_analyzer's Introduction

EML analyzer

EML analyzer is an application to analyze the EML file which can:

Analyze headers.
Analyze bodies.
- Extract IOCs (URLs, domains, IP addresses, emails) in bodies.
Analyze attachments.
- Identify whether attachments contain suspicious OLE files.

Installation

Docker

git clone https://github.com/ninoseki/eml_analyzer.git
cd eml_analyzer
docker build . -t eml_analyzer
docker run -i -d -p 8000:8000 eml_analyzer

The application is running at: http://localhost:8000/ in your browser.

Docker Compose

git clone https://github.com/ninoseki/eml_analyzer.git
cd eml_analyzer
docker-compose up

Docker vs. Docker compose

Docker:
- Run Uvicorn and SpamAssassin in the same container. (The processes are managed by Circus)
Docker Compose:
- Run Gunicorn and SpamAssassin in each container.

Thus Docker Compose is suitable for the production use.

Heroku

Alternatively, you can deploy the application on Heroku.

Configuration

Configuration can be done via environment variables.

Alternatively you can set values through .env file. Values in .env file will be automatically loaded.

Key	Desc.	Default
`INQUEST_API_KEY`	InQuest API key	-
`REDIS_EXPIRE`	Redis cache expiration time (in seconds)	3600
`REDIS_KEY_PREFIX`	Redis key prefix	`analysis`
`REDIS_URL`	Redis URL	-
`SPAMASSASSIN_HOST`	SpamAssassin host	`127.0.0.1`
`SPAMASSASSIN_PORT`	SpamAssassin port	783
`SPAMASSASSIN_TIMEOUT`	SpamAssassin timeout (in seconds)	10
`URLSCAN_API_KEY`	urlscan.io API Key	-
`VIRUSTOTAL_API_KEY`	VirusTotal API Key	-
`ASYNC_MAX_AT_ONCE`	Max number of concurrently running lookup tasks	`None`
`ASYNC_MAX_PER_SECOND`	Max number of tasks spawned per second	`None`

ToDo

Support MSG format.
In-depth attachments analysis by using oletools.

eml_analyzer's People

Contributors

Stargazers

Watchers

Forkers

casimkhan 3453-315h actorexpose ytreister 0xfork bigbrobro dekoder n1f2c3 bhagyeshkumar reuttetris r00t0vi4 hr-bit 5l1v3r1 yoonlee-lab namjatiffk jimwangzx rtvw4nqw tapsykrett wornet-aer hartl3y94 saloninteractive cosrah gbica-hzo jesusgavancho arcy24 dnkk28 towfiq007 paul-taylor-tp paulstasiuk rc042 silocityit avgirl stephan-rz pawnstorm0x kraken8585 nazywam blue-infosec cvlabsio gmh5225 darkleono

eml_analyzer's Issues

Typo on analyzer output page: 'Extracted emalis' should be 'Extracted emails'

Custom links on specific email headers

In a mailserver setup with search console it would be quite cool to have custom short links on specific email headers.

E.g. After loading an EML into the analyzer I'd like to:

view all emails received from this sender on my mailserver web console
view the mailserver logs for this message-id in elasticsearch/logstash
view all emails with the same subject to other recipients

For example in iRedMail a search URL looks like this: iredmail.example.com/activities/received/user/[email protected]. So it would be easily possible to construct a dynamic link pointing to the search results. But I guess this feature could also work with other email server solutions like mailcow or modoboa.

Unfortunately I'm not yet familiar with Vue.js and have no idea where to start when implementing such a feature.
Please let me know what you think.

Cheers
Andreas

RFI: Allow Attachment Download

Love the tool, figured I'd suggest adding an option to download extracted attachments for further (manual) analysis; not just upload them to Inquest/VT.
Cheers!

【Feature Request】support zip file upload

Why not consider adding compressed file uploads later, analyzing multiple .eml files in batches, and supporting historical query viewing. Your project is great, I gave you stars.

Uploading to VirusTotal

Where do I add my VT API key so that I can upload attachments to VT?

[Chrore] Website link

Seems the link is is dead

and has to be replaced https://eml-analyzer.up.railway.app/ -> https://eml-analyzer.herokuapp.com/

[Feature Request] Add link to Browserling for Extracted URLs

Hi,
Thank you so much for opensourcing this analyzer , it is really great for email analysis and I'm looking forward to further updates :) . A quick feature request : Could there be an option to add Browserling : https://www.browserling.com/ that will execute a URL ?
Example for google.com, the url will be https://www.browserling.com/browse/win/7/ie/11/http%3A%2F%2Fgoogle.com

SSL

I have a commercial cert, how can I use it in the docker image for https?

Cannot change source port for eml analyzer

Hello,

I am using portainer which already has the host port 8000 allocated. I tried changing all port values from your tool: 8000 -> 8005 but the tool does not spin up after this. Any advice?

GitHub Issue: EML Analyzer - Error in Date Parsing

Description:

After cloning the main branch of the EML Analyzer repository and adding the VirusTotal API key, I encountered an issue while running the Docker container. The application seems to start successfully, but when making a POST request to analyze an EML file, a 500 Internal Server Error occurs. The error trace indicates a problem in the date parsing process.

Steps to Reproduce:

Clone the main branch of the EML Analyzer repository.
Add the VirusTotal API key to the appropriate environment variable.
Run the application using docker-compose up.
Make a POST request to analyze an EML file.

Expected Behavior:

The application should successfully analyze the EML file and return the appropriate response.

Actual Behavior:

The application returns a 500 Internal Server Error, and the error trace indicates a TypeError related to date parsing.

Error Trace:

...
eml_analyzer_1  | TypeError: Cannot parse argument of type None.
...

Environment:

Operating System: Ubuntu 22.04.3
Docker Version: [Docker version 20.10.18, build b40c2f6]

Additional Information:
VirusTotal API Key has been added to the environment variables.
The issue seems to be related to date parsing in the EML file processing.
It happens with several eml and msg files, it seems unrelated to the Target.

Note:

I have verified that this issue occurs consistently in the provided environment. Any assistance in resolving this issue would be highly appreciated.

Internal Server Error

attempting to load https://raw.githubusercontent.com/oblac/jodd/master/jodd-mail/src/test/resources/jodd/mail/test/example.eml
onto demo https://eml-analyzer.herokuapp.com

Enabling SSL Support

Hi,

How do we enable SSL for the deployed instance?

URL follwing should be added

URL following should be added to trace the actual URL destination after HTTP redirections

Docker image creation error

Hello, when creating the image docker I have an error on the cmd sa-update

docker build . -t eml_analyzer

Thank you for your contribution and your help.

docker?

version? :)

Old header from value when checking multiple emails in a row

When analyzing multiple emails within the same "session" (without doing a browser reload), the Header-From value always contains the sender of the first email which has been uploaded.

After doing a short debugging dive, it looks like the from header value gets correctly reported by the backend as a response to the check request, but isn't refreshed within the frontend.

Can't copy extracted URL

Hello, I've tried to copy the extracted URL, but it looks that I only have the possibility to send it to VT or other providers. Sometimes the URLs contain sensitive info and I would like to remove it before uploading it to external scanners.
Is there any way to do that?

Thanks!

Email Upload Storage

Hi ninoseki,

Question,

Will the app store the email after analyzing it?

Not Working Anymore :(

Regards!

ReferenceError: regeneratorRuntime is not defined

Encountered the following error in the front end of the app:

vue.runtime.esm.js?2b0e:1897 ReferenceError: regeneratorRuntime is not defined
    at w (vue-concurrency.module.js?4b21:1:1)
    at setup (cjs.js?40c3:69:1)
    at mergedSetupFn (vue-composition-api.mjs?ed09:2160:1)
    at eval (vue-composition-api.mjs?ed09:1972:1)
    at activateCurrentInstance (vue-composition-api.mjs?ed09:1891:1)
    at initSetup (vue-composition-api.mjs?ed09:1970:1)
    at VueComponent.wrappedData (vue-composition-api.mjs?ed09:1953:1)
    at getData (vue.runtime.esm.js?2b0e:4761:1)
    at initData (vue.runtime.esm.js?2b0e:4718:1)
    at initState (vue.runtime.esm.js?2b0e:4655:1)

I was able to bypass it by installing the regenerator-runtime package and including it in the main.ts imports.

Is there something else going on that would cause that error?

Adding AnyRun to Attachment submission

Hi ninoseki,

Is it possible to add an option for AnyRun as well for file submission? Currently the 'Submit To' only supports Inquest and VT. Great tool btw!

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

chore(deps): update dependency eslint-plugin-prettier to v3.4.1

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

docker-compose

docker-compose.yml

instantlinux/spamassassin 3.4.6-1

dockerfile

Dockerfile

node 18-buster-slim

python 3.9-slim-buster

app.Dockerfile

node 18-buster-slim

python 3.9-slim-buster

github-actions

.github/workflows/deploy.yml

actions/checkout v2

akhileshns/heroku-deploy v3.12.12

.github/workflows/node.yml

actions/checkout v3

actions/setup-node v3

.github/workflows/test.yml

actions/checkout v3

abatilo/actions-poetry v2

actions/setup-python v4

python 3-slim

npm

frontend/package.json

@fortawesome/fontawesome-free 6.2.1

@mdi/font 7.0.96

@vue/composition-api ^1.7.1

@vueuse/core ^9.6.0

axios 1.2.0

buefy 0.9.22

core-js ^3.26.1

dayjs 1.11.6

filesize.js 2.0.0

highlight.js 11.6.0

js-base64 3.7.3

js-file-download ^0.4.12

js-sha256 0.9.0

qs 6.11.0

regenerator-runtime ^0.13.11

url-parse 1.5.10

vue 2.6.14

vue-concurrency 2.2.1

vue-markdown 2.2.4

vue-router 3.5.2

@types/jest 27.5.1

@types/js-base64 3.3.1

@types/qs 6.9.7

@types/url-parse 1.4.8

@typescript-eslint/eslint-plugin 4.29.0

@typescript-eslint/parser 4.29.0

@vue/cli-plugin-babel 4.5.13

@vue/cli-plugin-eslint 4.5.13

@vue/cli-plugin-typescript 4.5.13

@vue/cli-plugin-unit-jest 4.5.13

@vue/cli-service 4.5.13

@vue/eslint-config-prettier 6.0.0

@vue/eslint-config-typescript 7.0.0

@vue/test-utils 1.2.2

eslint 6.8.0

eslint-plugin-prettier 3.4.0

eslint-plugin-simple-import-sort 7.0.0

eslint-plugin-vue 7.19.1

prettier 2.7.1

typescript 4.7.3

vue-template-compiler 2.6.14

poetry

pyproject.toml

python ^3.9

aiofiles ^0.8.0

aiometer ^0.3.0

aiospamc ^0.9.0

arrow ^1.2.3

async-timeout ^4.0.2

beautifulsoup4 ^4.11.1

compoundfiles ^0.3

compressed-rtf ^1.0.6

dateparser ^1.1.2

eml_parser 1.17.5

fastapi ^0.85.2

fastapi-utils ^0.2.1

gunicorn ^20.1.0

html2text ^2020.1.16

httpx ^0.23.0

ioc-finder ^6.0.1

loguru ^0.6.0

oletools 0.60.1

pydantic ^1.10.2

python-multipart ^0.0.5

uvicorn ^0.19.0

vt-py ^0.17.1

aioresponses ^0.7.3

autoflake ^1.7

autopep8 ^2.0.0

black ^22.10.0

coveralls ^3.3.1

flake8 ^5.0.4

isort ^5.10.1

mypy ^0.982

pre-commit ^2.20.0

pytest ^7.2.0

pytest-asyncio ^0.20.1

pytest-cov ^4.0.0

pytest-env ^0.8.1

pytest-mock ^3.10.0

pytest-parallel ^0.1.1

pytest-randomly ^3.12.0

pytest-sugar ^0.9.5

pytest-timeout ^2.1.0

pyupgrade ^3.2.0

respx ^0.20.0

vcrpy ^4.2.1

py ^1.11.0

Check this box to trigger a request for Renovate to run again on this repository

Is there a file size limit?

When uploading a 15 megabytes large eml file, I'm getting an empty response from the API.

The POST request to /api/analyze/file is sent, but after some time 0 B are returned and the loading circle in the GUI just disappears.

Please let me know if you need further details.

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Internal Server Error

I have tried different .eml and .msg but every time i see a Internal server error

SpamAssassin index error during parsing body details

I'm encountering a parsing error at line:

eml_analyzer/backend/clients/spamassasin.py

Line 47 in a756fdb

if "---" in line:

When I've tested the code with the following sample email:

GTUBE = """Subject: Test spam mail (GTUBE)
Message-ID: <[email protected]>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <[email protected]>
To: Recipient <[email protected]>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

--- This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments! --- 

This is the GTUBE, the
    Generic
    Test for
    Unsolicited
    Bulk
    Email

If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should send this test mail from an account outside of your network.
""".encode(
    "ascii"
)

The code breaks as the sample contains --- This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments! --- in the body.

Using ---- instead of --- resolves the issue.

for index, line in enumerate(lines):
    if "----" in line:
        delimiter_index = index + 1
        break

Unable to extract URL from eml content

Hey,

In section Extracted URLs, there are options for url lookup to VirusTotal, UrlScan.io and Browserling. So, the eml that has following content url,

Following was the result of extracted URL which is not proper query format for mentioned URL scanner. That is, the marked values are considered as the part of embedded url.

For eg.

Thank You.