@jonluca
Anubis version: none (cloned repo today)
Python version: 'Python 3.6.5rc1'
Machine Details:
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 02)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 620 (rev 02)
00:04.0 Signal processing controller: Intel Corporation Skylake Processor Thermal Subsystem (rev 02)
00:14.0 USB controller: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller (rev 21)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-LP Thermal subsystem (rev 21)
00:15.0 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #0 (rev 21)
00:15.1 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #1 (rev 21)
00:16.0 Communication controller: Intel Corporation Sunrise Point-LP CSME HECI #1 (rev 21)
00:17.0 SATA controller: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] (rev 21)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port (rev f1)
00:1c.4 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #5 (rev f1)
00:1c.5 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-LP LPC Controller (rev 21)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-LP PMC (rev 21)
00:1f.3 Audio device: Intel Corporation Sunrise Point-LP HD Audio (rev 21)
00:1f.4 SMBus: Intel Corporation Sunrise Point-LP SMBus (rev 21)
01:00.0 3D controller: NVIDIA Corporation GM108M [GeForce 940MX] (rev a2)
02:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

O.S.: Linux parrot 4.16.0-parrot5-amd64 #1 SMP Parrot 4.16.5-1parrot5 (2018-05-03) x86_64 GNU/Linux

Full Stack Trace:
https://pastebin.com/gzdyuaVn

Cryptography requirements:

sudo apt-get install build-essential libssl-dev libffi-dev python3-dev
build-essential is already the newest version (12.4).
libffi-dev is already the newest version (3.2.1-8).
libssl-dev is already the newest version (1.1.0h-2).
python3-dev is already the newest version (3.6.4-1).

Features:

Please note by far the quickest way to get a new feature is to make a Pull Request.

Otherwise, open an issue and tag it with "feature-request"

The solution

We are suggesting you add Spyse as one of the data sources for: DNS, subdomain enumeration, retrieving web information, hosts, certificates, and ports.
Potential is immense. Contact us if you need any help or a free subscription to make it free for users.

Possible alternatives

Alternatives could be shodan or censys.

Additional context

I'm offering Spyse because it has different scanning and analyzing systems that provide a more clear and sometimes larger amounts of data.

Notice: I work for Spyse.

A future project might be to recreate a scan on all the IPs of the subdomains, or pass them off to nessus.

Not able to get Shodan result because of the installation error.

Command "/usr/bin/python3.4 -u -c "import setuptools, tokenize;file='/tmp/pip-build-t8955aq8/cryptography/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-i5krkv2s-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-t8955aq8/cryptography/

Please follow the general troubleshooting steps first:

• Check you are using Python 3.6
• Make sure you are using pip3 (or any instance of pip associated with Python 3.x)

Bug reports:

Some subdomains in the result contain *.:

vpn.awscorp.uberinternal.com
corpmdm.uberinternal.com
*.uberinternal.com
vpn-china-szx1.uberinternal.com
--- snippet ---
jira-cdn-staging.uberinternal.com
*.tm1ext.awscorp.uberinternal.com
uberhubtest.uberinternal.com
gw08-usw1.awscorp.uberinternal.com
--- snippet ---
lsfosq_us1.uberinternal.com
*.it-tools.uberinternal.com
gw08-sae1.awscorp.uberinternal.com

I've never gotten this to install properly. Has there been any thought into deploying this as a Snap or Flatpak? I would hope a universal installer would eliminate the particulars of dependency problems and steps involved to set up in a mixed python environment. I'm one of those casual Linux users that gets lost once steps get too far in the weeds with pip installing things

┌──(kali㉿Kali)-[~/Anubis]
└─$ sudo apt-get install python3-pip python-dev libssl-dev libffi-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package python-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
python2-dev python2 python-dev-is-python3

E: Package 'python-dev' has no installation candidate

I want to know the name of a man using fake names

In readme.md it reads:

API Keys
If you wish to use the --aditional-info flag, you'll need to set up an API key for https://shodan.io.
Copy API_SAMPLE.py to APY.py,

You might want to change the last line to:

Copy API_SAMPLE.py to API.py,

mind the 'I' instead of the 'Y'.

Installing collected packages: anubis-netsec
Successfully installed anubis-netsec-0.2.0

~# anubis
Traceback (most recent call last):
File "/usr/local/bin/anubis", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.5/dist-packages/anubis/cli.py", line 78, in main
import anubis.commands
File "/usr/local/lib/python3.5/dist-packages/anubis/commands/init.py", line 1, in
from .target import *
File "/usr/local/lib/python3.5/dist-packages/anubis/commands/target.py", line 26, in
from anubis.utils.ColorPrint import ColorPrint
ImportError: No module named 'anubis.utils'

#OS: MacOS 10.12.6
#Python: 2.7.14
Error like this:

Traceback (most recent call last):
File "/usr/local/bin/anubis", line 11, in
load_entry_point('Anubis', 'console_scripts', 'anubis')()
File "/opt/information-gathering/subdomains/Anubis/anubis/cli.py", line 73, in main
import anubis.commands
File "/opt/information-gathering/subdomains/Anubis/anubis/commands/init.py", line 1, in
from .target import *
File "/opt/information-gathering/subdomains/Anubis/anubis/commands/target.py", line 9, in
from urllib.parse import urlsplit
ImportError: No module named parse

Anubis (0.1.0, /opt/information-gathering/subdomains/Anubis)
asn1crypto (0.23.0)
attrs (17.3.0)
backports-abc (0.5)
censys (0.0.8)
certifi (2017.11.5)
cffi (1.11.2)
chardet (3.0.4)
click (6.7)
click-plugins (1.0.3)
colorama (0.3.9)
configparser (3.5.0)
coverage (4.4.2)
cryptography (1.9)
Django (1.11.7)
dnspython (1.15.0)
docopt (0.6.2)
enum34 (1.1.6)
funcsigs (1.0.2)
gevent (1.2.2)
greenlet (0.4.12)
idna (2.6)
ipaddress (1.0.18)
lxml (4.1.1)
nassl (1.0.1)
netaddr (0.7.19)
parse (1.8.2)
pdfkit (0.6.1)
pip (9.0.1)
pluggy (0.6.0)
py (1.5.2)
pyasn1 (0.4.2)
pycparser (2.18)
pyOpenSSL (17.5.0)
pytest (3.3.1)
pytest-cov (2.5.1)
python-nmap (0.6.1)
pytz (2017.3)
requests (2.10.0)
rsa (3.4.2)
setuptools (36.5.0)
shodan (1.7.7)
singledispatch (3.4.0.3)
six (1.11.0)
SSLyze (1.2.0)
tls-parser (1.1.1)
tornado (4.5.2)
typing (3.6.2)
urllib3 (1.22)
wheel (0.30.0)
XlsxWriter (1.0.2)

https://github.com/KOLANICH/RichConsole

Please follow the general troubleshooting steps first:

• [ Yes] Check you are using Python 3.6
• [ Yes] Make sure you are using pip3 (or any instance of pip associated with Python 3.x)

Bug reports:

Make sure you include the following:

• Anubis Version - 0.6.4
• Python Version - 3.6.3
• Full stack trace if crash or installation error

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 142, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/util/connection.py", line 91, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/util/connection.py", line 81, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 351, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 814, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 254, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 151, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10f246828>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 623, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/util/retry.py", line 281, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.pkey.in', port=443): Max retries exceeded with url: /tools-i/search-subdomains (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10f246828>: Failed to establish a new connection: [Errno 60] Operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/anubis", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/anubis/cli.py", line 115, in main
    command.run()
  File "/usr/local/lib/python3.6/site-packages/anubis/commands/target.py", line 76, in run
    Thread(target=search_pkey(self, self.options["TARGET"][i])),
  File "/usr/local/lib/python3.6/site-packages/anubis/scanners/pkey.py", line 18, in search_pkey
    headers=headers, data=data, verify=False)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 111, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.pkey.in', port=443): Max retries exceeded with url: /tools-i/search-subdomains (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10f246828>: Failed to establish a new connection: [Errno 60] Operation timed out',))

• Machine details (OS, build number, distro, etc.)

macOS 10.13.2

It looks like https://pkey.in times out which causes urllib3 to get a timeout exception. Once this exception occurs the scan times out and quits. I suggest removing pkey or creating an easy way to disable that specific searcher through a command line flag. Additionally there may need to be additional error handling to recover from these issues.

Note: pkey.in is also timing out in my browser.

Please follow the general troubleshooting steps first:

• Check you are using Python 3.6
• Make sure you are using pip3 (or any instance of pip associated with Python 3.x)

Bug reports:

Make sure you include the following:

• Anubis Version 1.1.3
• Python Version 3.11.8
• Full stack trace if crash or installation error N/A
• Machine details (OS, build number, distro, etc.) Pop!_OS 22.04 LTS

The default scan for anubis includes searching sublist3r but gives an error: Exception when searching sublist3r. Looking at the code:

results = requests.get('https://api.sublist3r.com/search.php', headers=headers, params=params)

It's calling to api.sublist3r.com, but the sublist3r.com domain name is now owned by some random domain parking company (pinging it resolves to lb01.parklogic.com).

I would suggest removing sublist3r as a resolution tool in Anubis.

P.S. Fantastic tool! Thanks so much!

@jonluca
I would like to suggest reading domains from file to make it easier to process multiple domains at once.
Sadly i cant code to do it on my own to suggest it via PR.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency dnspython to v2.6.1 [SECURITY]
• Update dependency requests to v2.32.2 [SECURITY]
• Update dependency censys to v2.2.12
• Update dependency shodan to v1.31.0
• Update python Docker tag to v3.12
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

• Anubis Version anubis_netsec-0.7.6-py2.py3-none-any.whl
• Python Version 3.6.3
• Full stack trace if crash or installation error
• Machine details (OS, build number, distro, etc.)
  ubuntu mate 17.10

@jonluca
not quite sure how to do a stack trace

pip3 install anubis-netsec
... downloads all the files...
Successfully built censys SSLyze docopt python-nmap shodan tls-parser click-plugins pycparser
Installing collected packages: certifi, urllib3, chardet, idna, requests, netaddr, censys, dnspython, setuptools, nassl, asn1crypto, six, pycparser, cffi, cryptography, tls-parser, SSLyze, docopt, python-nmap, click, click-plugins, colorama, XlsxWriter, shodan, anubis-netsec
Successfully installed SSLyze-1.3.4 XlsxWriter-1.0.2 anubis-netsec-0.7.6 asn1crypto-0.24.0 censys-0.0.8 certifi-2018.1.18 cffi-1.11.4 chardet-3.0.4 click-6.7 click-plugins-1.0.3 colorama-0.3.9 cryptography-2.1.4 dnspython-1.15.0 docopt-0.6.2 idna-2.6 nassl-1.0.3 netaddr-0.7.19 pycparser-2.18 python-nmap-0.6.1 requests-2.18.4 setuptools-38.5.1 shodan-1.7.7 six-1.11.0 tls-parser-1.2.1 urllib3-1.22
Segmentation fault (core dumped)

kern.log
Feb 24 17:02:08 kernel: [ 311.672024] random: crng init done
Feb 24 17:02:14 kernel: [ 317.955766] pip3[1638]: segfault at a9 ip 0000000000442f24 sp 00007ffda779e648 error 4 in python3.6[400000+3bf000]
Feb 24 17:05:22 kernel: [ 505.137210] pip3[1932]: segfault at 22e728 ip 00000000004dc698 sp 00007ffca67b2a00 error 4 in python3.6[400000+3bf000]
Feb 24 17:17:56 kernel: [ 1259.481690] pip3[11814]: segfault at 22e728 ip 00000000004dc698 sp 00007ffe9eed6350 error 4 in python3.6[400000+3bf000]

syslog
Feb 24 17:17:56 kernel: [ 1259.481690] pip3[11814]: segfault at 22e728 ip 00000000004dc698 sp 00007ffe9eed6350 error 4 in python3.6[400000+3bf000]

Hello

Currently, there is no way to get only subdomains and not logo and debug messages.
So add --silent or just pipe debug to stderr, so you can do ’anubis -t substack.com 2>/dev/null’

@jonluca

• Check you are using Python 3.6
• Make sure you are using pip3 (or any instance of pip associated with Python 3.x)

Bug reports:

Make sure you include the following:

• Anubis Version = 1.0.3
• Python Version = 3.9
• Full stack trace if crash or installation error
  Error connecting to target! Make sure you spelled it correctly and it is a resolvable address Traceback (most recent call last): File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python39\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Python39\Scripts\anubis.exe\__main__.py", line 7, in <module> File "C:\Python39\lib\site-packages\anubis\cli.py", line 127, in main command.run() File "C:\Python39\lib\site-packages\anubis\commands\target.py", line 68, in run self.init() File "C:\Python39\lib\site-packages\anubis\commands\target.py", line 64, in init raise e File "C:\Python39\lib\site-packages\anubis\commands\target.py", line 59, in init domain_str = socket.gethostbyname(host) socket.gaierror: [Errno 11001] getaddrinfo failed
• Machine details (OS, build number, distro, etc.) = Windows 10

Tried to enumerate a list of domains with Anubis in Powershell.
Tried both:
example.com
https://example.com

To confirm the issue I asked a friend and he gets the same error. So Im assuming its code or format of the list related not a missing dependency.

Consider adding Hudson Rock's complimentary data to receive additional URLs associated with a specific domain.
Data was retrieved from infected computers with the Infostealer malware and contains URLs that usually cannot be found conventionally.

Domain sample: https://cavalier.hudsonrock.com/api/json/v2/osint-tools/search-by-domain?domain=tesla.com

Thank you.

Companion issue to jonluca/Anubis-DB#1

readme.md reads:

pip install -r requirements

While the filename is requirements.txt

Suggestion: since Anubis is running using Python 3.6 and I'm running both Python2 and Python3, I had to specify pip3 in order to get the requirements installed. (macos 10.13.2)

WARNING: The script anubis is installed in '/home/kali/.local/bin' which is not on PATH.
this is showing ............please tell me what to do next......

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

• set url to "https://github.com/jonluca/Anubis.git"

Evidence:
" git clone [email protected]:jonluca/Anubis.git
Cloning into 'Anubis'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists."

When cloning the repo on a small bandwith connection I wondered about the relative transfer size vs. the working tree size. When preparing the snap in #22 you must've accidentally added to and then later removed the package blob from the repository. I think there is an argument for a quick clone, even without --depth 1. Removing the snap and a python binary reduces the size from 19M to 3M, and there are other artifacts too.

I can recommend the oneliner at stackoverflow.com#42544963 to list the biggest blobs and than the java util bfg to remove by size or path. It's an easier frontend than git filter-branch.

Disadvantage of the removal is the rewrite of commit hashes since the introduction of each blob. bfg will output a graph for this. People having forked from one of these commits would need to hard reset their forks and rebase. In my opinion this is not an issue with a smaller codebase where most people will prepare a PR from the latest origin/HEAD.

git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| sed -n 's/^blob //p' \
| sort --numeric-sort --key=2 \
| cut -c 1-12,41- \
| $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest \
| grep -vF --file=<(git ls-tree -r HEAD | awk '{print $3}') \
| tac | head -n20

827f720a141a   15MiB anubis_0.8.3+git1.cbabacf_amd64.snap
5d2ef13a6db4  4,3MiB prime/usr/bin/python3.5
cbbc03be80b6  1,4MiB anubis/common_subdomains.txt
6fc2f1710815  117KiB prime/usr/share/doc/python3.5/NEWS.gz

java -jar bfg.jar --delete-files 'anubis_0.8.3+git1.cbabacf_amd64.snap'
java -jar bfg.jar --delete-files 'python3.5'

Earliest                                              Latest
|                                                          |
...................................................Dmmmmmmmm

D = dirty commits (file tree fixed)
m = modified commits (commit message or parents changed)
. = clean commits (no changes to file tree)

Down at 3344K.

optional:

java -jar bfg.jar --delete-files 'common_subdomains.txt'
java -jar bfg.jar --delete-folders 'prime'
java -jar bfg.jar --delete-folders 'stage'
java -jar bfg.jar --delete-files 'temp.txt'

after those deletes:

git reflog expire --expire=now --all && git gc --prune=now --aggressive
du -cs .

With all deletes, the repo is down at 540K.

Please follow the general troubleshooting steps first:

• Check you are using Python 3.6
• Make sure you are using pip3 (or any instance of pip associated with Python 3.x)

Bug reports:

Make sure you include the following:

• Anubis Version
• Python Version
• Full stack trace if crash or installation error
• Machine details (OS, build number, distro, etc.)

Also, please assign @jonluca to all issues at first.

Please replace this line with a brief summary of your issue.

Features:

Please note by far the quickest way to get a new feature is to make a Pull Request.

Otherwise, open an issue and tag it with "feature-request"

==============================================================

I'm trying to install into docker alpine image.
I tried python versions 3.6.12, 3.7.9, 3.8.5

installation command tried python -m pip install --no-cache-dir anubis-netsec
also tried to build from source with pip3 install .

ERROR: Could not find a version that satisfies the requirement sslyze==3.0.7 (from anubis-netsec) (from versions: 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, ERROR: No matching distribution found for sslyze==3.0.7 (from anubis-netsec)

For a sneak peek, check out the recursive_search_on_subdomain branch

I've done a substantial reorganization of the code such that all the functions are now modules in anubis.scanners. This'll make it trivial to add new ones, and will prevent bloat in target.py.

The feature will be merged into master in the next few days.