joshhighet / bebop Goto Github PK

in addition to shodan for external lookups, allow for queries to the binaryedge.io dataset

internetdb.shodan.io

in addition to shodan for external lookups, allow for queries to the censys.io dataset

continue analysis against matches when sub-processors return findings

an idea - needs to be fleshed out.. some confidence-scoring system will likely required before aimlessly probing potential matches

this could include things such as;

• if etag match found, correlate against other identifiers on the target to raise confidence
• using hidden service address as host header to the discovered asset (same for referrer)
• enrich found assets with pdns history from the likes of securitytrails

discover hidden paths through bruteforcing and other tricks.

see joshhighet/kerchow/sbin/http-scanner, gobuster & dirsearch

if a Last-Modified header value exists, span out a search for it across any supported search services (shodan etc)

⚠️ this may have too many fp's and value should be validated first

per below service.onion apparently matches conditions for all of the checks. in reality - it's not detecting the catchall and these are incorrect.

INFO     127:configcheck.py  fetch                     found 200 at http://service.onion/sitemap.xml
INFO     127:configcheck.py  fetch                     found 200 at http://service.onion/admin
INFO     127:configcheck.py  fetch                     found 200 at http://service.onion/administrator
INFO     127:configcheck.py  fetch                     found 200 at http://service.onion/wp-admin
INFO     127:configcheck.py  fetch                     found 200 at http://service.onion/.env
INFO     127:configcheck.py  fetch                     found 200 at http://service.onion/WEB-INF/web.xml

avoiding potential false flags i.e service.onion below

➜  bebop git:(main) ✗ uuidgen
C27362AF-4A1C-4E82-918F-FB6CC89E6EB8
➜  bebop git:(main) ✗ curl --socks5-hostname telemetry.dark:9050 -X GET -I http://service.onion/C27362AF-4A1C-4E82-918F-FB6CC89E6EB8
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 03 Aug 2023 10:25:03 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 423
Last-Modified: Wed, 02 Aug 2023 10:18:39 GMT
Connection: keep-alive
Accept-Ranges: bytes

• detect subdomain wildcarding
• detect URI-path wildcarding

add fofa as an available engine to the collection of search processors

I give the api keys of shodan, zoomeye and other sites but it didn't accepted ......if you could tell me the correct way to sync them it would a big help for me...

scan libraries and correlate to known outdated libraries and provide further context.

see retire.js

provide an option to leverage an alternative to the python requests library that supports DOM rendering

ahem undetected-chromedriver

if a site is serving a certificate, leverage said elements against subprocessors

also consider jarm

use

• virustotal
• securitytrails
• urlscan

then validate resolution

potentially encoding related

➜  app git:(main) ✗ python3 subprocessors.py
DEBUG:__main__:zoomeye: querying title:'Fox Trot'
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.zoomeye.org:443
DEBUG:urllib3.connectionpool:https://api.zoomeye.org:443 "GET /host/search?query=title%3A%27Fox+Trot%27 HTTP/1.1" 500 94
ERROR:__main__:zoomeye: api error: 500 Server Error: Internal Server Error for url: https://api.zoomeye.org/host/search?query=title%3A%27Fox+Trot%27

➜  app git:(main) ✗ curl -vv \
'https://api.zoomeye.org/host/search?query=title:"Fox%20Trot"' -H "API-KEY:[stripped]"

*   Trying 160.116.186.18:443...
* Connected to api.zoomeye.org (160.116.186.18) port 443 (#0)
[stripped]
> 
< HTTP/1.1 500 Internal Server Error
[stripped]
< 
* Connection #0 to host api.zoomeye.org left intact
{"error": "internal_error", "message": "internal error", "url": "https://www.zoomeye.org/api"}%  

alongside nmap's service identification, gain a better understanding of frameworks and technologies on a site.

see wappalyzer

in addition to shodan for external lookups, allow for queries to the zoomeye.org dataset

Zoomeye is using the old API endpoints. The new HK URL’s should be used

once enough confidence is gained that the method of identifying 'rare' headers is functional, consider searching said values against the likes of shodan

provide the ability to return screenshots of paged which have undergone review

something like the playwright implementation seen here ransomwatch/screenshotter.py