qeeqbox / url-sandbox Goto Github PK

View Code? Open in Web Editor NEW

164.0 9.0 51.0 4.43 MB

Scalable URL Sandbox for analyzing URLs and Domains from phishing attacks

License: GNU Affero General Public License v3.0

Python 69.99% HTML 26.45% Shell 1.37% JavaScript 2.19%

url sandbox phishing domain malware ads analyzer analyzing-urls python docker

url-sandbox's People

Contributors

Stargazers

Watchers

Forkers

chaitanyakrishna sashka3076 jurya digitalarche itsintern 5l1v3r1 lee2020d s3bp2021 jdbarnett1 actorexpose jimwangzx e4lra ctfbuster igor-2021 ob2010 ekmixon c1x1x00xxpentium ua-solsen ghbook joe-2022 samml99 dulestar rdsbackup spartanbb kaganisildak crackercat pianomanx vicw4r bazaaaaaaaar tadryanom fyodorr mc2099 sts0mrg0 king29988 fabiomb17 mmilleror hashtables2 lamthanhngan jinjiao070 croat79 chudry lac-amar ilir4734 l-amar-test r3dc4t0x00 arshadsiddiquie soufrancisco kali973 le-ton-nhan xoxma1993 brainhub24

url-sandbox's Issues

Docker issue

Getting below error when running any URL scan and job does not finish, followed standard installation of project. Would appreciate any assistance, thank you.

url-sandbox-master-mongodb-1 | 2023-03-06T10:12:43.674+1000 I ACCESS [conn7] Successfully authenticated as principal changeme_9620eh26sfvka017fx on admin
url-sandbox-master-workers_api-1 | [2023-03-06 10:12:46,724: WARNING/ForkPoolWorker-1] Error -> 404 Client Error for http+docker://localhost/v1.39/images/create?tag=latest&fromImage=url-sandbox-box: Not Found ("pull access denied for url-sandbox-box, repository does not exist or may require 'docker login'") 45be86aa-b217-4803-92f3-805b7392ee6e
url-sandbox-master-workers_api-1 | [2023-03-06 10:12:46,729: ERROR/ForkPoolWorker-1] Task analyze_url[e59c4473-462b-44e3-a30d-cfc9cb0bbf87] raised unexpected: FileNotFoundError(2, 'No such file or directory')
url-sandbox-master-workers_api-1 | Traceback (most recent call last):
url-sandbox-master-workers_api-1 | File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 451, in trace_task
url-sandbox-master-workers_api-1 | R = retval = fun(*args, **kwargs)
url-sandbox-master-workers_api-1 | File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 734, in protected_call
url-sandbox-master-workers_api-1 | return self.run(*args, **kwargs)
url-sandbox-master-workers_api-1 | File "/backend/worker.py", line 85, in analyze_url
url-sandbox-master-workers_api-1 | make_report(parsed)
url-sandbox-master-workers_api-1 | File "/backend/qbreport.py", line 160, in make_report
url-sandbox-master-workers_api-1 | analyzer_db = TinyDB(analyzer_path)
url-sandbox-master-workers_api-1 | File "/usr/local/lib/python3.7/site-packages/tinydb/database.py", line 94, in init
url-sandbox-master-workers_api-1 | self._storage: Storage = storage(*args, **kwargs)
url-sandbox-master-workers_api-1 | File "/usr/local/lib/python3.7/site-packages/tinydb/storages.py", line 102, in init
url-sandbox-master-workers_api-1 | touch(path, create_dirs=create_dirs)
url-sandbox-master-workers_api-1 | File "/usr/local/lib/python3.7/site-packages/tinydb/storages.py", line 31, in touch
url-sandbox-master-workers_api-1 | with open(path, 'a'):
url-sandbox-master-workers_api-1 | FileNotFoundError: [Errno 2] No such file or directory: '/output/45be86aa-b217-4803-92f3-805b7392ee6e-analyzer.logs'

Docker
Docker version 18.09.7, build 2d0083d
OS
Linux qeeqbox-VirtualBox 4.15.0-29-generic #31~16.04.1-Ubuntu SMP Wed Jul 18 08:54:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

installation takes too much time

Hi, im trying to install url-sandbox using your script on a ubuntu desktop 20 lts with the latest updates, and its been a day and a half and still hangging on "pip is looking at multiple versions of python-dateutil to determine which version is compatible with other requirements. This could take a while. Collecting python-dateutil>=2.7" , if i restart it installs python's lib all over again , is there any script which speeds up the installation?, i really do want to put this to work , best regards.

Build Error

Environment:
OS: Ubuntu 22.04
Arch: amd64
Kernel: 5.15.0-48-generic
Docker:

Client: Docker Engine - Community
 Version:           20.10.18
 API version:       1.41
 Go version:        go1.18.6
 Git commit:        b40c2f6
 Built:             Thu Sep  8 23:11:43 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.18
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.6
  Git commit:       e42327a
  Built:            Thu Sep  8 23:09:30 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Error:

url-sandbox-proxy-1        | Oct 08 23:10:35.000 [notice] Bootstrapped 10% (conn_done): Connected to a relay
url-sandbox-proxy-1        | Oct 08 23:10:35.000 [notice] Bootstrapped 14% (handshake): Handshaking with a relay
url-sandbox-proxy-1        | Oct 08 23:10:36.000 [notice] Bootstrapped 15% (handshake_done): Handshake with a relay done
url-sandbox-proxy-1        | Oct 08 23:10:36.000 [notice] Bootstrapped 20% (onehop_create): Establishing an encrypted directory connection
url-sandbox-proxy-1        | Oct 08 23:10:36.000 [notice] Bootstrapped 25% (requesting_status): Asking for networkstatus consensus
url-sandbox-box-1          | /usr/local/lib/python3.7/site-packages/scapy/layers/ipsec.py:471: CryptographyDeprecationWarning: Blowfish has been deprecated
url-sandbox-box-1          |   cipher=algorithms.Blowfish,
url-sandbox-box-1          | /usr/local/lib/python3.7/site-packages/scapy/layers/ipsec.py:485: CryptographyDeprecationWarning: CAST5 has been deprecated
url-sandbox-box-1          |   cipher=algorithms.CAST5,
url-sandbox-proxy-1        | Oct 08 23:10:36.000 [notice] Bootstrapped 30% (loading_status): Loading networkstatus consensus
url-sandbox-proxy-1        | Oct 08 23:10:36.000 [notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
url-sandbox-proxy-1        | Oct 08 23:10:36.000 [notice] Bootstrapped 40% (loading_keys): Loading authority key certs
url-sandbox-proxy-1        | Oct 08 23:10:37.000 [notice] The current consensus has no exit nodes. Tor can only build internal paths, such as paths to onion services.
url-sandbox-proxy-1        | Oct 08 23:10:37.000 [notice] Bootstrapped 45% (requesting_descriptors): Asking for relay descriptors
url-sandbox-proxy-1        | Oct 08 23:10:37.000 [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/6521, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of end bw (no exits in consensus, using mid) = 0% of path bw.)
url-sandbox-proxy-1        | Oct 08 23:10:37.000 [notice] Bootstrapped 50% (loading_descriptors): Loading relay descriptors
url-sandbox-box-1 exited with code 0
url-sandbox-workers_api-1  | Traceback (most recent call last):
url-sandbox-workers_api-1  |   File "/usr/local/bin/celery", line 8, in <module>
url-sandbox-workers_api-1  |     sys.exit(main())
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/__main__.py", line 14, in main
url-sandbox-workers_api-1  |     from celery.bin.celery import main as _main
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/bin/celery.py", line 18, in <module>
url-sandbox-workers_api-1  |     from celery.app.utils import find_app
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/app/__init__.py", line 2, in <module>
url-sandbox-workers_api-1  |     from celery import _state
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/_state.py", line 15, in <module>
url-sandbox-workers_api-1  |     from celery.utils.threads import LocalStack
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/utils/__init__.py", line 16, in <module>
url-sandbox-workers_api-1  |     from .nodenames import nodename, nodesplit, worker_direct
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/utils/nodenames.py", line 6, in <module>
url-sandbox-workers_api-1  |     from kombu.entity import Exchange, Queue
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/kombu/entity.py", line 7, in <module>
url-sandbox-workers_api-1  |     from .serialization import prepare_accept_content
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/kombu/serialization.py", line 440, in <module>
url-sandbox-workers_api-1  |     for ep, args in entrypoints('kombu.serializers'):  # pragma: no cover
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/kombu/utils/compat.py", line 82, in entrypoints
url-sandbox-workers_api-1  |     for ep in importlib_metadata.entry_points().get(namespace, [])
url-sandbox-workers_api-1  | AttributeError: 'EntryPoints' object has no attribute 'get'
url-sandbox-website-1      | [2022-10-08 23:10:38 +0200] [1] [INFO] Starting gunicorn 20.1.0
url-sandbox-website-1      | [2022-10-08 23:10:38 +0200] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
url-sandbox-website-1      | [2022-10-08 23:10:38 +0200] [1] [INFO] Using worker: gevent
url-sandbox-website-1      | [2022-10-08 23:10:38 +0200] [10] [INFO] Booting worker with pid: 10
url-sandbox-proxy-1        | Oct 08 23:10:38.000 [notice] The current consensus contains exit nodes. Tor can build exit and internal paths.
url-sandbox-workers_api-1 exited with code 1
url-sandbox-proxy-1        | Oct 08 23:10:39.000 [notice] Bootstrapped 56% (loading_descriptors): Loading relay descriptors
url-sandbox-proxy-1        | Oct 08 23:10:39.000 [notice] Bootstrapped 63% (loading_descriptors): Loading relay descriptors
url-sandbox-website-1      | [2022-10-08 23:10:39 +0200] [10] [ERROR] Exception in worker process
url-sandbox-website-1      | Traceback (most recent call last):
url-sandbox-website-1      |   File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
url-sandbox-website-1      |     worker.init_process()
url-sandbox-website-1      |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 146, in init_process
url-sandbox-website-1      |     super().init_process()
url-sandbox-website-1      |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 134, in init_process
url-sandbox-website-1      |     self.load_wsgi()
url-sandbox-website-1      |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
url-sandbox-website-1      |     self.wsgi = self.app.wsgi()
url-sandbox-website-1      |   File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
url-sandbox-website-1      |     self.callable = self.load()
url-sandbox-website-1      |   File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
url-sandbox-website-1      |     return self.load_wsgiapp()
url-sandbox-website-1      |   File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
url-sandbox-website-1      |     return util.import_app(self.app_uri)
url-sandbox-website-1      |   File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 359, in import_app
url-sandbox-website-1      |     mod = importlib.import_module(module)
url-sandbox-website-1      |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
url-sandbox-website-1      |     return _bootstrap._gcd_import(name[level:], package, level)
url-sandbox-website-1      |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
url-sandbox-website-1      |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
url-sandbox-website-1      |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
url-sandbox-website-1      |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
url-sandbox-website-1      |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
url-sandbox-website-1      |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
url-sandbox-website-1      |   File "/website/web.py", line 34, in <module>
url-sandbox-website-1      |     from celery import Celery
url-sandbox-website-1      | ImportError: cannot import name 'Celery' from 'celery' (/usr/local/lib/python3.7/site-packages/celery/__init__.py)
url-sandbox-website-1      | [2022-10-08 23:10:40 +0200] [10] [INFO] Worker exiting (pid: 10)
url-sandbox-proxy-1        | Oct 08 23:10:40.000 [notice] Bootstrapped 68% (loading_descriptors): Loading relay descriptors
url-sandbox-website-1      | [2022-10-08 23:10:40 +0200] [1] [INFO] Shutting down: Master
url-sandbox-website-1      | [2022-10-08 23:10:40 +0200] [1] [INFO] Reason: Worker failed to boot.
url-sandbox-proxy-1        | Oct 08 23:10:41.000 [notice] Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits
url-sandbox-proxy-1        | Oct 08 23:10:41.000 [notice] Bootstrapped 80% (ap_conn): Connecting to a relay to build circuits
url-sandbox-proxy-1        | Oct 08 23:10:41.000 [notice] Bootstrapped 85% (ap_conn_done): Connected to a relay to build circuits
url-sandbox-proxy-1        | Oct 08 23:10:41.000 [notice] Bootstrapped 89% (ap_handshake): Finishing handshake with a relay to build circuits
url-sandbox-proxy-1        | Oct 08 23:10:41.000 [notice] Bootstrapped 90% (ap_handshake_done): Handshake finished with a relay to build circuits
url-sandbox-proxy-1        | Oct 08 23:10:41.000 [notice] Bootstrapped 95% (circuit_create): Establishing a Tor circuit
url-sandbox-proxy-1        | Oct 08 23:10:42.000 [notice] Bootstrapped 100% (done): Done
url-sandbox-website-1 exited with code 3
url-sandbox-workers_api-1  | Traceback (most recent call last):
url-sandbox-workers_api-1  |   File "/usr/local/bin/celery", line 8, in <module>
url-sandbox-workers_api-1  |     sys.exit(main())
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/__main__.py", line 14, in main
url-sandbox-workers_api-1  |     from celery.bin.celery import main as _main
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/bin/celery.py", line 18, in <module>
url-sandbox-workers_api-1  |     from celery.app.utils import find_app
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/app/__init__.py", line 2, in <module>
url-sandbox-workers_api-1  |     from celery import _state
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/_state.py", line 15, in <module>
url-sandbox-workers_api-1  |     from celery.utils.threads import LocalStack
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/utils/__init__.py", line 16, in <module>
url-sandbox-workers_api-1  |     from .nodenames import nodename, nodesplit, worker_direct
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/utils/nodenames.py", line 6, in <module>
url-sandbox-workers_api-1  |     from kombu.entity import Exchange, Queue
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/kombu/entity.py", line 7, in <module>
url-sandbox-workers_api-1  |     from .serialization import prepare_accept_content
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/kombu/serialization.py", line 440, in <module>
url-sandbox-workers_api-1  |     for ep, args in entrypoints('kombu.serializers'):  # pragma: no cover
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/kombu/utils/compat.py", line 82, in entrypoints
url-sandbox-workers_api-1  |     for ep in importlib_metadata.entry_points().get(namespace, [])
url-sandbox-workers_api-1  | AttributeError: 'EntryPoints' object has no attribute 'get'
url-sandbox-workers_api-1 exited with code 1
url-sandbox-workers_api-1  | Traceback (most recent call last):
url-sandbox-workers_api-1  |   File "/usr/local/bin/celery", line 8, in <module>
url-sandbox-workers_api-1  |     sys.exit(main())
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/__main__.py", line 14, in main
url-sandbox-workers_api-1  |     from celery.bin.celery import main as _main
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/bin/celery.py", line 18, in <module>
url-sandbox-workers_api-1  |     from celery.app.utils import find_app
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/app/__init__.py", line 2, in <module>
url-sandbox-workers_api-1  |     from celery import _state
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/_state.py", line 15, in <module>
url-sandbox-workers_api-1  |     from celery.utils.threads import LocalStack
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/utils/__init__.py", line 16, in <module>
url-sandbox-workers_api-1  |     from .nodenames import nodename, nodesplit, worker_direct
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/celery/utils/nodenames.py", line 6, in <module>
url-sandbox-workers_api-1  |     from kombu.entity import Exchange, Queue
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/kombu/entity.py", line 7, in <module>
url-sandbox-workers_api-1  |     from .serialization import prepare_accept_content
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/kombu/serialization.py", line 440, in <module>
url-sandbox-workers_api-1  |     for ep, args in entrypoints('kombu.serializers'):  # pragma: no cover
url-sandbox-workers_api-1  |   File "/usr/local/lib/python3.7/site-packages/kombu/utils/compat.py", line 82, in entrypoints
url-sandbox-workers_api-1  |     for ep in importlib_metadata.entry_points().get(namespace, [])
url-sandbox-workers_api-1  | AttributeError: 'EntryPoints' object has no attribute 'get'
url-sandbox-workers_api-1 exited with code 1

Error 500

I get error 500 both when trying to login to 127.0.0.1:8000 or trying to register

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.