Code Monkey home page Code Monkey logo

uncover's Introduction


Quickly discover exposed hosts on the internet using multiple search engines.

FeaturesInstallationUsageConfigurationRunning UncoverJoin Discord


uncover is a go wrapper using APIs of well known search engines to quickly discover exposed hosts on the internet. It is built with automation in mind, so you can query it and utilize the results with your current pipeline tools.

Features

httpx

Installation Instructions

uncover requires go1.20 to install successfully. Run the following command to get the repo -

go install -v github.com/projectdiscovery/uncover/cmd/uncover@latest

Usage

uncover -h

This will display help for the tool. Here are all the flags it supports:

Usage:
  ./uncover [flags]

Flags:
INPUT:
   -q, -query string[]   search query, supports: stdin,file,config input (example: -q 'example query', -q 'query.txt')
   -e, -engine string[]  search engine to query (shodan,shodan-idb,fofa,censys,quake,hunter,zoomeye,netlas,criminalip,publicwww,hunterhow) (default shodan)

SEARCH-ENGINE:
   -s, -shodan string[]       search query for shodan (example: -shodan 'query.txt')
   -sd, -shodan-idb string[]  search query for shodan-idb (example: -shodan-idb 'query.txt')
   -ff, -fofa string[]        search query for fofa (example: -fofa 'query.txt')
   -cs, -censys string[]      search query for censys (example: -censys 'query.txt')
   -qk, -quake string[]       search query for quake (example: -quake 'query.txt')
   -ht, -hunter string[]      search query for hunter (example: -hunter 'query.txt')
   -ze, -zoomeye string[]     search query for zoomeye (example: -zoomeye 'query.txt')
   -ne, -netlas string[]      search query for netlas (example: -netlas 'query.txt')
   -cl, -criminalip string[]  search query for criminalip (example: -criminalip 'query.txt')
   -pw, -publicwww string[]   search query for publicwww (example: -publicwww 'query.txt')
   -hh, -hunterhow string[]   search query for hunterhow (example: -hunterhow 'query.txt')

CONFIG:
   -pc, -provider string         provider configuration file (default "$CONFIG/uncover/provider-config.yaml")
   -config string                flag configuration file (default "$CONFIG/uncover/config.yaml")
   -timeout int                  timeout in seconds (default 30)
   -rl, -rate-limit int          maximum number of http requests to send per second
   -rlm, -rate-limit-minute int  maximum number of requests to send per minute
   -retry int                    number of times to retry a failed request (default 2)

OUTPUT:
   -o, -output string  output file to write found results
   -f, -field string   field to display in output (ip,port,host) (default "ip:port")
   -j, -json           write output in JSONL(ines) format
   -r, -raw            write raw output as received by the remote api
   -l, -limit int      limit the number of results to return (default 100)
   -nc, -no-color      disable colors in output

DEBUG:
   -silent   show only results in output
   -version  show version of the project
   -v        show verbose output

Using uncover as library

Example of using uncover as library is provided in examples directory.

Provider Configuration

The default provider configuration file should be located at $CONFIG/uncover/provider-config.yaml and has the following contents as an example.

Note: API keys are required and must be configured before running uncover.

shodan:
  - SHODAN_API_KEY_1
  - SHODAN_API_KEY_2
censys:
  - CENSYS_API_ID_1:CENSYS_API_SECRET_1
  - CENSYS_API_ID_2:CENSYS_API_SECRET_2
fofa:
  - FOFA_EMAIL_1:FOFA_KEY_1
  - FOFA_EMAIL_2:FOFA_KEY_2
quake:
  - QUAKE_TOKEN_1
  - QUAKE_TOKEN_2
hunter:
  - HUNTER_API_KEY_1
  - HUNTER_API_KEY_2
zoomeye:
  - ZOOMEYE_API_KEY_1
  - ZOOMEYE_API_KEY_2
netlas:
  - NETLAS_API_KEY_1
  - NETLAS_API_KEY_2
criminalip:
  - CRIMINALIP_API_KEY_1
  - CRIMINALIP_API_KEY_2
publicwww:
  - PUBLICWWW_API_KEY_1
  - PUBLICWWW_API_KEY_2
hunterhow:
  - HUNTERHOW_API_KEY_1 
  - HUNTERHOW_API_KEY_2

When multiple keys/credentials are specified for same provider in the config file, random key will be used for each execution.

alternatively you can also set the API key as environment variable in your bash profile.

export SHODAN_API_KEY=xxx
export CENSYS_API_ID=xxx
export CENSYS_API_SECRET=xxx
export FOFA_EMAIL=xxx
export FOFA_KEY=xxx
export QUAKE_TOKEN=xxx
export HUNTER_API_KEY=xxx
export ZOOMEYE_API_KEY=xxx
export NETLAS_API_KEY=xxx
export CRIMINALIP_API_KEY=xxx
export PUBLICWWW_API_KEY=xxx
export HUNTERHOW_API_KEY=xxx

Required API keys can be obtained by signing up on following platform Shodan, Censys, Fofa, Quake, Hunter, ZoomEye, Netlas, CriminalIP and Publicwww.

Running Uncover

Default run:

uncover supports multiple ways to make the query including stdin or q flag, as default shodan engine is used for search if no engine is specified.

echo 'ssl:"Uber Technologies, Inc."' | uncover 
                                        
  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.9    
                                        

		projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[WRN] By using uncover, you also agree to the terms of the APIs used.

107.180.12.116:993
107.180.26.155:443
104.244.99.31:443
161.28.20.79:443
104.21.8.108:443
198.71.233.203:443
104.17.237.13:443
162.255.165.171:443
12.237.119.61:443
192.169.250.211:443
104.16.251.50:443

Running uncover with file input containing multiple search queries per line.

cat dorks.txt

ssl:"Uber Technologies, Inc."
title:"Grafana"
uncover -q dorks.txt
                                        
  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.9    
                                        

    projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[WRN] By using uncover, you also agree to the terms of the APIs used.

107.180.12.116:993
107.180.26.155:443
104.244.99.31:443
161.28.20.79:443
104.21.8.108:443
198.71.233.203:443
2607:7c80:54:3::74:3001
104.198.55.35:80
46.101.82.244:3000
34.147.126.112:80
138.197.147.213:8086

Single query against multiple search engine

uncover supports multiple search engine, as default shodan is used, -e flag can be used to run same query against any or all search engines.

echo jira | uncover -e shodan,censys,fofa,quake,hunter,zoomeye,netlas,criminalip

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.9  
                                        

    projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[WRN] By using uncover, you also agree to the terms of the APIs used.

176.31.249.189:5001
13.211.116.80:443
43.130.1.221:631
192.195.70.29:443
52.27.22.181:443
117.48.120.226:8889
106.52.115.145:49153
13.69.135.128:443
193.35.99.158:443
18.202.109.218:8089
101.36.105.97:21379
42.194.226.30:2626

Multiple query against multiple search engine

uncover -shodan 'http.component:"Atlassian Jira"' -censys 'services.software.product=`Jira`' -fofa 'app="ATLASSIAN-JIRA"' -quake 'Jira' -hunter 'Jira' -zoomeye 'app:"Atlassian JIRA"' -netlas 'jira' -criminalip 'Jira'

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.9
                                        

    projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[WRN] By using uncover, you also agree to the terms of the APIs used.

104.68.37.129:443
162.222.160.42:443
34.255.84.133:443
52.204.121.166:443
23.198.29.120:443
136.156.180.95:443
54.194.233.15:443
104.117.55.155:443
149.81.4.6:443
54.255.218.95:443
3.223.137.57:443
83.228.124.171:443
23.202.195.82:443
52.16.59.25:443
18.159.145.227:443
104.105.53.236:443

Shodan-InternetDB API

uncover supports shodan-internetdb API to pull available ports for given IP/CIDR input.

shodan-idb used as default engine when IP/CIDR is provided as input, otherwise shodan search engine is used.

echo 51.83.59.99/24 | uncover

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.9  
                                        

    projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[WRN] By using uncover, you also agree to the terms of the APIs used.

51.83.59.1:53
51.83.59.1:10000
51.83.59.2:53
51.83.59.3:25
51.83.59.3:80
51.83.59.3:389
51.83.59.3:443
51.83.59.3:465
51.83.59.3:587
51.83.59.3:993

Field Format

-f, -field flag can be used to indicate which fields to return, currently, ip, port, and host are supported and can be used to return desired fields.

uncover -q jira -f host -silent

ec2-44-198-22-253.compute-1.amazonaws.com
ec2-18-246-31-139.us-west-2.compute.amazonaws.com
tasks.devrtb.com
leased-line-91-149-128-229.telecom.by
74.242.203.213.static.inetbone.net
ec2-52-211-7-108.eu-west-1.compute.amazonaws.com
ec2-54-187-161-180.us-west-2.compute.amazonaws.com
185-2-52-226.static.nucleus.be
ec2-34-241-80-255.eu-west-1.compute.amazonaws.com

Field Formatting

uncover has a -f, -field flag that can be used to customize the output format. For example, in the case of uncover -f https://ip:port/version, ip:port will be replaced with results in the output while keeping the format defined, It can also be used to specify a known scheme/path/file in order to prepare the output so that it can be immediately passed as input to other tools in the pipeline.

echo kubernetes | uncover -f https://ip:port/version -silent

https://35.222.229.38:443/version
https://52.11.181.228:443/version
https://35.239.255.1:443/version
https://34.71.48.11:443/version
https://130.211.54.173:443/version
https://54.184.250.232:443/version

Output of uncover can be further piped to other projects in workflow accepting stdin as input, for example:

  • uncover -q example -f ip | naabu - Runs naabu for port scanning on the found host.
  • uncover -q title:GitLab | httpx - Runs httpx for web server probing the found result.
  • uncover -q 51.83.59.99/24 | httpx - Runs httpx on host/ports obtained from shodan-internetdb.
uncover -q http.title:GitLab -silent | httpx -silent

https://15.185.150.109
https://139.162.137.16
https://164.68.115.243
https://135.125.215.186
https://163.172.59.119
http://15.236.10.197
https://129.206.117.248
  • uncover -q 'org:"Example Inc."' | httpx | nuclei - Runs httpx / nuclei for vulnerability assessment.

image

Notes:

  • keys/ credentials are required to configure before running or using this project.
  • query flag supports all and only filters supported by search engine.
  • results are limited to 100 as default and can be increased with limit flag.
  • shodan-idb API doesn't requires an API key and works out of the box.
  • shodan-idb API is used as default engine when IP/CIDR is provided as input.

uncover is made with 🖤 by the projectdiscovery team.

uncover's People

Contributors

chen2aaron avatar codgin avatar dependabot[bot] avatar dogancanbakir avatar edoardottt avatar ehsandeep avatar gy741 avatar ice3man543 avatar jakewarren avatar luitelsamikshya avatar metzinaround avatar mzack9999 avatar parrasajad avatar ramanareddy0m avatar shubhamrasal avatar tarunkoyalwar avatar xc1ym avatar xm1k3 avatar zmf963 avatar zt2 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

uncover's Issues

parse env vars: missing censys secret

echo sonylive.com | uncover  -e shodan,fofa,censys -v

 __  ______  _________ _   _____  _____
/ / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.2  

   	projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
By using uncover, you also agree to the terms of the APIs used.

[WRN] couldn't parse env vars: missing censys secret

When calling FOFA API, the set field name is wrong

https://github.com/projectdiscovery/uncover/blob/main/uncover/agent/fofa/fofa.go#L17

I guess you are planning to use domain field to represent the result domain name information, but in FOFA domain field represents the primary domain name of the target.

Example, when the result is: api.baidu.com, then domain value of the field is baidu.com. If a user wants to query all Baidu domain names in FOFA, your way of writing will only return one record baidu.com.

图片

图片

You should use the host field in the FOFA API to represent domain information.

reduce the number of requests

According to the uncover document, you can search for multiple dorks

My idea is to read multiple statements in the dork.txt file, which can be combined together, and one request can get the return results of all statements

dork.txt

ssl:"Uber Technologies, Inc."
title:"Grafana"

For example, all engines support || or flag

fofa :  ssl:"Uber Technologies, Inc." || title:"Grafana"
censys : ip="1.1.1.1" or  ip="1.1.1.1"

Originally posted by @Lengso in #10

Unexpected EOF for Larger Limit

uncover version:

Current Behavior:

[shodan] unexpected EOF when we use larger limit like 20000

Expected Behavior:

expected behavior it should completed the scan getting 20000 hosts

Steps To Reproduce:

  1. uncover -q 'title:"xfinity"' -v -l 20000

  2. image

Anything else:

Uncover not writing output.

root@m3An-MAcHIn3:/dsk# uncover -q 'nginx' -o test.txt


  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.8

                projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
By using uncover, you also agree to the terms of the APIs used.

184.72.249.120:80
34.69.144.173:80
192.0.78.12:443
151.101.2.133:443
35.231.23.183:80
...

root@m3An-MAcHIn3:/dsk# cat test.txt | wc -l
0

Uncover is not writing results to file

360 Quake - limited output

There seems to be a bug in relation to the number of results which can be extracted by uncover. Without -l it is not 100 but 300. With -l being provided it is still limited to 500 max. Even if you set -l 5000 , it won't go beyond 500. Now, the actual Quake website allows to browse and download 10 000's of results.

Search queries with commas (`,`) get splitted into multiple queries when using Shodan

uncover version

be23df8 compiled using go version devel go1.19-5bb2628c6f darwin/amd64

Current Behavior

$ cat $HOME/.config/uncover/config.yaml 
query:
  - 'org:"Something, Inc."' 

engine: shodan
...

Results in the following queries issues by the tool:

[shodan] unexpected status code 502 received from https://api.shodan.io/shodan/host/search?key=REDACTED&query=org:"Something&page=1
[shodan] unexpected status code 502 received from https://api.shodan.io/shodan/host/search?key=REDACTED&query= Inc."&page=1

Expected Behavior

A single query is issued using my raw string, urlencoding it if needed.

Steps To Reproduce:

$ SHODAN_API_KEY=REPLACE-WITH-YOURS uncover -q 'org:"Something, Inc."' -v

Anything else

For now I've fixed this by doing the needed changes in order for the program to handle Query parameter as a string type rather than goflags.FileCommaSeparatedStringSlice. Please, note this required more changes but I avoided pasting them there.

diff --git a/runner/options.go b/runner/options.go
index 17804d6..a23b7e5 100644
--- a/runner/options.go
+++ b/runner/options.go
@@ -23,7 +23,7 @@ var (
 
 // Options contains the configuration options for tuning the enumeration process.
 type Options struct {
-       Query        goflags.FileCommaSeparatedStringSlice
+       Query        string

Would be happy to contribute a change to use Query as a string, but maybe the proper fix is to make goflags.FileCommaSeparatedStringSlice properly handle quotted commas? I'll leave that decision to your team! 😃

SHODAN_API_KEY_2 is used as a key

Took me some time to figure out why shodan module wasn't working... So I added API key 1 because I only have one key, but I did not delete - SHODAN_API_KEY_2 from provider-config.yaml
For some reason, uncover decided not to use the real key and just kept trying to use SHODAN_API_KEY_2 as a key. Once I removed
- SHODAN_API_KEY_2 everything started working fine.

Support for Shodan internetdb API

Discussed in #13

Originally posted by blacklist-arcc March 7, 2022
It would be nice to have a list of ip addresses as input instead of queries in order to quickly identify open ports on specific targets from various engines instead of a port scan running by naabu for example. Maybe there is an option already using a specific search string e.g. ip:1.1.1.1

In this case a command line parameter "prepend" may be a smooth solution in order to prepend a specific string to every line that comes from std in order to get a pipeline like:

subfinder -d TARGET | dnsx -resp-only -a | uncover | httpx | nuclei -t cve

Hopefully I have not overseeing something :)

Cheers.

IP and CIDR (using mapcidr) input needs to be supported.

echo 51.83.59.99/24 | uncover

51.83.59.99:22
51.83.59.99:80
51.83.59.99:443
51.83.59.99:500
echo 51.83.59.99 | uncover -json


{
  "timestamp":1648469130,
  "source":"shodan-internetdb",
  "ip":"5.9.248.1",
  "ports":[
    22,
    80,
    443,
    500
  ],
  "cpes":[
    "cpe:/a:igor_sysoev:nginx",
    "cpe:/a:openbsd:openssh:7.4"
  ],
  "host":[
    "www.sampleresponse.fr"
  ],
  "tags":[
    "vpn"
  ],
  "vulns":[
    "CVE-2017-15906"
  ]
}

As the default shodan search API gets used, need to confirm if IP/CIDR input can be determined to use shodan-idb instead, otherwise, engine name needs to be specified to use the new source.

Reference: https://internetdb.shodan.io

panic: interface conversion: interface {} is string, not float64 (zoomeye)

Running from dev | main

./uncover -zoomeye 'title:"gogs"'
94.103.206.187:443
110.40.191.14:443
110.42.188.156:443
110.42.193.145:443
128.199.39.72:8443
92.242.45.237:443
119.45.254.3:443
202.164.44.246:3000
panic: interface conversion: interface {} is string, not float64

goroutine 20 [running]:
github.com/projectdiscovery/uncover/uncover/agent/zoomeye.(*Agent).query(0x0?, {0x100bafdde?, 0x0?}, 0x0?, 0x0?, 0x0?)
	/Users/geekboy/Github/uncover/uncover/agent/zoomeye/zoomeye.go:104 +0x3c0
github.com/projectdiscovery/uncover/uncover/agent/zoomeye.(*Agent).Query.func1()
	/Users/geekboy/Github/uncover/uncover/agent/zoomeye/zoomeye.go:51 +0xd0
created by github.com/projectdiscovery/uncover/uncover/agent/zoomeye.(*Agent).Query
	/Users/geekboy/Github/uncover/uncover/agent/zoomeye/zoomeye.go:40 +0x134

integration test for supported providers

Please describe your feature request:

Currently, there is no integration test for supported search engines/providers that can be implemented.

  • Shodan
  • Censys
  • Fofa
  • Quake
  • ZoomEye

Integration test will be using ENV variable to read API information.

support for delay between request

Based on censys docs it's has rate limited

Screenshot_2022-03-06_03_42_20

so while I used uncover I got too many requests so it's possible to add delay between requests when using censys engine to bypass rate limited , plus I think uncover doesn't randomize multiple tokens because I used one token and got 12K result then I got too many requests so I used multiple tokens and got the same result , I think if uncover randomize the multiple tokens we can bypass the rate limited

RATE-LIMIT:
   -delay int                    delay in seconds between requests to the same host (default 1)

Censys - Add option to include include Virtual Hosts

Currently the Censys agent exclude all Virtual Hosts:

URL = "https://search.censys.io/api/v2/hosts/search?q=%s&per_page=%d&virtual_hosts=EXCLUDE"

I think those should be included or there should be an option to include them if needed.

This won't return any results:

echo 'name: "*.arvala7.space" or name: "*.kintan.space"' | uncover -e censys -v

but the same query on Censys with Virtual Hosts included will return 7 results.

`-l` argument does not work when engine is fofa

uncover version:

1.0.1

Current Behavior:

/ # uncover -q 'cert="baidu" && (icon_hash="-1374555452" || icon_hash="1684500512")' -l 2000 -json -o /tmp/result.json -e fofa
/ # wc -l /tmp/result.json
91 /tmp/result.json

-l 2000 return little record, more fewer than 2000.

besides, the search result is bigger than 2000.

image

Expected Behavior:

-l 2000 return 2000 result

Steps To Reproduce:

/ # uncover -q 'cert="baidu" && (icon_hash="-1374555452" || icon_hash="1684500512")' -l 2000 -json -o /tmp/result.json -e fofa

Anything else:

HunterHow - invalid memory address or nil pointer dereference

I'm getting the following when trying to run hunterhow:

$/home/kali/go/bin/uncover -q hunterhow.query -e hunterhow -l 1000 -o hunterhow.txt

[INF] Current uncover version v1.0.3 (latest)
[hunterhow] couldn't create new session: no default rate limit found for engine hunterhow
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xc9cbd8]

goroutine 36 [running]:
github.com/projectdiscovery/uncover/uncover/agent/hunterhow.(*Agent).Query(0xc0004b5350?, 0xe3f9d9?, 0xdd7d93?)
        /home/kali/go/pkg/mod/github.com/projectdiscovery/[email protected]/uncover/agent/hunterhow/hunterhow.go:24 +0x18
github.com/projectdiscovery/uncover/runner.(*Runner).Run.func1({0x1031a70, 0x162eaf0}, 0xc00040e450?)
        /home/kali/go/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:184 +0x4af
created by github.com/projectdiscovery/uncover/runner.(*Runner).Run
        /home/kali/go/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:165 +0x1cdb

Am I missing something? Do I need to edit something?
Also, there's no HunterHow's key section in the provider-config.yaml file.

`-r` and `-j` option is is not working on default run

uncover version:

v1.0.5 (latest)

Current Behavior:

When running the command 'uncover.exe -r -q "any query"' in the latest version (v1.0.5), I encounter a problem.

Expected Behavior:

I expected the command to execute and provide the desired output.

Steps To Reproduce:

  1. Run 'uncover.exe -r -q "any query"' command.

Anything else:

I have noticed that the latest version (v1.0.5) don't produce any output with -j and -r flags
without -r or -j it works normally. but i need json for scripting.

uncover.exe -r -q element

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /
\__,_/_/ /_/\___/\____/|___/\___/_/

                projectdiscovery.io

[INF] Current uncover version v1.0.5 (latest)

uncover.exe -j -q element

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /
\__,_/_/ /_/\___/\____/|___/\___/_/

                projectdiscovery.io

[INF] Current uncover version v1.0.5 (latest)

with 1.0.3

uncover.exe -r -q element

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /
\__,_/_/ /_/\___/\____/|___/\___/_/

                projectdiscovery.io

[INF] Current uncover version v1.0.3 (outdated)
{"_shodan":{"crawler": .....

shodan struggles with multiple queries

In the queries file for shodan I have about 100 similar requests. I can cram them all into a single query and shodan works (slowly) just fine but if brake them up into shorter queries, the module struggles:

image
As you can see, shodan returns 503 and 429.

# number of times to retry a failed request
retry: 10

^^^
I have edited the config file to try to address the issue but the edit seems to be irrelevant in this particular case. Basically, in order to extract all results I have to pack multiple requests into a single one.

CriminalIP - can't handle multiple queries

I have 70 queries in the file and nothing goes through.

The site responds with

[criminalip] unexpected status code 500 received from https://api.criminalip.io/v1/banner/search?offset=1&query=

quake api is not working on windows

=> uncover.exe -qk 'google.com' -v -config /provider-config.yaml

=> [�[33mquake�[0m] json: cannot unmarshal string into Go struct field Response.code of type int

api key is 100% correct and working

-o doesn't work in 1.0.4

I can see the results in the terminal, however, the output is not saved and there's no output file at all.

add file check before reading provider config

uncover version:

main | dev

Current Behavior:

  • issue detected in nuclei projectdiscovery/nuclei#3749
  • when $HOME/.config/uncover does not exist . fileutil.Unmarshal treats it as a string and unmarshal its which causes unmarshal error

Expected Behavior:

no error

Steps to Reproduce

  1. Temporarily move config folder to different location
mv ~/.config/uncover ~/.config/uncoverold
  1. Run Nuclei it will show unmarshal error
$ nuclei
[ERR] failed to load provider keys got yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `/Users/...` into sources.Provider

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.5

		projectdiscovery.io

[WRN] Found 3 templates loaded with deprecated protocol syntax, update before v2.9.5 for continued support.
[INF] Current nuclei version: v2.9.5 (latest)
[INF] Current nuclei-templates version: v9.5.0 (latest)
[INF] New templates added in latest release: 62
[INF] Templates loaded for current scan: 5961
^C[INF] CTRL+C pressed: Exiting

^ this happens because we try to unmarshal string

Anything else:

publicwww - new api provider

Please describe your feature request:

New provider source - https://publicwww.com
API endpoint and details - https://publicwww.com/profile/api (Public signup with API is available)

Supported field

  • host (existing)
  • url (new)
echo "/wp-content/themes/twentysixteen/" | uncover -silent -e publicwwww -f url

https://www.mcmaster.ca
https://winaero.com
https://getitfree.us
https://semioffice.com
https://www.saberia.com
https://bezux.pl
http://thespacereporter.com
https://www.cbrc.jp
https://obrnadzor.gov.ru
https://www.infokids.gr
https://techmixx.de
....
echo "/wp-content/themes/twentysixteen/" | uncover -silent -e publicwwww -f host

www.mcmaster.ca
winaero.com
getitfree.us
semioffice.com
www.saberia.com
bezux.pl
thespacereporter.com
www.cbrc.jp
obrnadzor.gov.ru
....

How to use configuration files

How to use configuration files?

image

documentation means
shodan:

  • SHODAN_API_KEY1
  • SHODAN_API_KEY2
    censys:
  • CENSYS_API_ID:CENSYS_API_SECRET
    fofa:
  • FOFA_EMAIL:FOFA_KEY

but my yaml file is
shodan: []
censys: []
fofa: []

Can you give a test example?thiks

Add support for retries

Please describe your feature request:

A server timeout may result in Shodan returning no results for a query. There should be an argument -retries int similar to httpx to retry the query if the response is unexpected and the default value set to 1.

CLI flag to add:

   -retries int                number of times to retry a failed request (default 1)

401 received

wt1

Hi , the commands in the image do not work , censys commands do not work either but this works:
echo jira | uncover -e shodan

Add Netlas Search Engine

Please describe your feature request:

Add netlas (https://netlas.io/) as search engine for uncover.

Describe the use case of this feature:

Add -ne/-netlas []string to search for queries on netlas.

ZoomEye - If added togheter with other engines, sometimes fail with the same error

When performing multiple searches using the -e zoomeye command along with other search engines, an error occasionally occurs with the following panic message:

panic: interface conversion: interface {} is string, not float64
goroutine 58 [running]:
github.com/projectdiscovery/uncover/uncover/agent/zoomeye.(*Agent).query(0x0?, {0x1286f8b?, 0x0?}, 0x0?, 0x0?, 0x0?)
	/home/runner/work/uncover/uncover/uncover/agent/zoomeye/zoomeye.go:94 +0x4e5
github.com/projectdiscovery/uncover/uncover/agent/zoomeye.(*Agent).Query.func1()
	/home/runner/work/uncover/uncover/uncover/agent/zoomeye/zoomeye.go:42 +0x10b
created by github.com/projectdiscovery/uncover/uncover/agent/zoomeye.(*Agent).Query
	/home/runner/work/uncover/uncover/uncover/agent/zoomeye/zoomeye.go:31 +0xcc

uncover version:

1.0.3

Steps To Reproduce:

-r -silent -e shodan,censys,zoomeye -q "apache"

Note that this issue occurs intermittently, not consistently with every search.

Ability to lookup/search query for specfic search engine

Please describe your feature request:

As of now all the query will be passed to all the engines if multiple engines that are specified (-e shodan,censys), instead for use case where different and specific query needs to be searched with specific engine is not possible.

pseudo example:

uncover -shodan "ssl:'Uber Inc'" -censys "ssl.dns:'Uber Inc'" -fofa "ssl = uber.com"

Describe the use case of this feature:

Search engine specific queries can be searched against multiple platform from same run.

From user:

I just configured uncover it works perfectly... but how can i use multiple query for particular search engine , like sodan uses different query than censys like sodan uses ssl:"program name" to search but censys uses a different one to search for the same query and same goes to other search engines .... is there any default query list set in uncover ...

The replace functionality within the `-field` flag is not working correctly

Taking the example from https://github.com/projectdiscovery/uncover#field-formatting

uncover -f https://ip:port/version

TODO:

  • Some escaping option needed: It is not possible to create a URL with a static path pointing to any of the reserved words (ip, host, port) e.g. http://10.10.10.10:1234/host
  • The replace logic also replaces substrings within different words: e.g. -f https://ip:port/hipster would also translate to something like https://10.10.10.10:1234/h10.10.10.10ster
  • Wrap placeholders with something like: {{ip}}

config issue

the /home/kali/.config/uncover/ dir doesn't exist to make the .yaml file

should I create one, will this work !!

Capture

normalized input query

uncover version:

0.0.1

Current Behavior:

uncover -q 'ssl:"Sony Global"' parsed as ssl:Sony+Global

Expected Behavior:

uncover -q 'ssl:"Sony Global"' parsed as ssl:"Sony+Global"

Refactor uncover and provide abstracted library (similar to tlsx)

Please describe your feature request:

The current implementation of uncover in nuclei is not abstract and is affected by minor changes, as noted in this GitHub issue (link to the issue). This is already a breaking change and we need to design a minimalistic Uncover API that abstracts everything.

As talked with @tarunKoyalwar, we create a new API that encapsulates all uncover-related functionality and expose it via a simple and stable interface. This will make it easier for developers to use uncover in their projects without worrying about the underlying implementation details.

Describe the use case of this feature:

By doing so, we can avoid having to update nuclei every time we add or update uncover.

refs: https://github.com/projectdiscovery/tlsx/blob/main/pkg/tlsx/tlsx.go

Receive domains as input for internetdb-api

Please describe your feature request:

Internal logic to resolve domain, use as input for internetdb, return all ports for all ips for that domain, ( projectdiscovery/naabu#293 )

echo google.com | uncover -json
{
  "timestamp":1648469130,
  "source":"shodan-internetdb",
  "ip":[
    "142.250.218.142",
    "142.250.218.141"
  ],
  "ports":[
    80,
    443
  ],
  "cpes":[
    "cpe:/a:igor_sysoev:nginx",
    "cpe:/a:openbsd:openssh:7.4"
  ],
  "host":[
    "google.com"
  ],
  "tags":[
    ""
  ],
  "vulns":[
    ""
  ]
}

Describe the use case of this feature:

subfinder -d TARGET | uncover -json | jq '.host[]+":"+(.ports[]|tostring)' |httpx

Improving `limit` option to display specified number of results

root@m3An-MAcHIn3:/dsk# uncover -q 'nginx' -o test.txt -limit 10 | wc -l

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.8

                projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
By using uncover, you also agree to the terms of the APIs used.

100
root@m3An-MAcHIn3:/dsk# uncover -q 'nginx' -o test.txt -limit 150 | wc -l

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.8

                projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
By using uncover, you also agree to the terms of the APIs used.

200
root@m3An-MAcHIn3:/dsk# uncover -q 'nginx' -o test.txt -limit 210 | wc -l

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /
\__,_/_/ /_/\___/\____/|___/\___/_/ v0.0.8

                projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
By using uncover, you also agree to the terms of the APIs used.

300

The limit logic is not working properly, For eg in the above case Shodan responds with 100 results per api call, Currently if uncover's limit parameter is < 100 then uncover will make single api request which will output 100 results, if the limit parameter is between 100-200 uncover will send 2 requests and show 200 results and so on, The proper logic should be as follows :

-limit 50 => Make single request to shodan's endpoint, 100 results retrieved => split top 50 results => output
-limit 150 => Make 2 requests to shodan's endpoint, 200 results retrieved => split top 150 results => output

and so on.

Error Installing Uncover

uncover version:

NA

Current Behavior:

While trying to install Uncover I am getting the following error:

image

Error:

github.com/modern-go/reflect2
gopkg.in/yaml.v3

github.com/modern-go/reflect2

go/pkg/mod/github.com/modern-go/[email protected]/unsafe_map.go:9:2: undefined: unsafeType
go/pkg/mod/github.com/modern-go/[email protected]/unsafe_link.go:18:55: undefined: sliceHeader
go/pkg/mod/github.com/modern-go/[email protected]/go_above_118.go:17:20: type2.rtype undefined (type *UnsafeMapType has no field or method rtype)
go/pkg/mod/github.com/modern-go/[email protected]/reflect2.go:145:22: undefined: unpackEFace
go/pkg/mod/github.com/modern-go/[email protected]/reflect2.go:157:22: undefined: unpackEFace
go/pkg/mod/github.com/modern-go/[email protected]/reflect2.go:168:14: undefined: safeType
go/pkg/mod/github.com/modern-go/[email protected]/reflect2.go:172:12: undefined: safeStructType
go/pkg/mod/github.com/modern-go/[email protected]/reflect2.go:174:10: undefined: newUnsafeStructType
go/pkg/mod/github.com/modern-go/[email protected]/reflect2.go:177:12: undefined: safeSliceType
go/pkg/mod/github.com/modern-go/[email protected]/reflect2.go:179:10: undefined: newUnsafeArrayType
go/pkg/mod/github.com/modern-go/[email protected]/reflect2.go:179:10: too many errors

gopkg.in/yaml.v3

go/pkg/mod/gopkg.in/[email protected]/decode.go:33:11: undefined: yaml_parser_t
go/pkg/mod/gopkg.in/[email protected]/decode.go:34:11: undefined: yaml_event_t
go/pkg/mod/gopkg.in/[email protected]/decode.go:35:12: undefined: Node
go/pkg/mod/gopkg.in/[email protected]/decode.go:36:23: undefined: Node
go/pkg/mod/gopkg.in/[email protected]/decode.go:80:27: undefined: yaml_event_type_t
go/pkg/mod/gopkg.in/[email protected]/decode.go:99:25: undefined: yaml_event_type_t
go/pkg/mod/gopkg.in/[email protected]/decode.go:140:28: undefined: Node
go/pkg/mod/gopkg.in/[email protected]/decode.go:147:27: undefined: Node
go/pkg/mod/gopkg.in/[email protected]/decode.go:314:11: undefined: Node
go/pkg/mod/gopkg.in/[email protected]/decode.go:315:15: undefined: Node
go/pkg/mod/gopkg.in/[email protected]/decode.go:147:27: too many errors

OS Info:
NAME=EndeavourOS
PRETTY_NAME=EndeavourOS
ID=endeavouros
ID_LIKE=arch

Not Wroking

uncover version:

Current Version: v0.0.5

Current Behavior:

When I run echo org:"Uber" | uncover -e shodan
or
uncover -e shodan -q 'org:"Uber"'

It doesn't show any result. I have checked my shodan key.
It works on shodan.io, just enter org:"Uber" in search.

Expected Behavior:

Where is the result?

Steps To Reproduce:

  1. Run uncover -e shodan -q 'org:"Uber"'
  2. See result

Fofa Error

Fofa can't get data(ips) fofa not work fofa not showing ips

Change rate limit logic and align the uncover flags the same way they are done on subfinder

Here are the messages covered also in this task: #157

Please describe your feature request:

#157 (comment)

from:

I think it will be better if we change ratelimit logic in uncover i.e in uncover we use delay to configure ratelimit while we have -rlm and -rl options in nuclei,subfinder,httpx . I think keeping options uniform will help rather than providing different formats for same functionality. Example

If source has ratelimit say 30 req/min

In nuclei we pass -rlm 30
In uncover we have to pass -delay 2

Apart from obvious calculation required another thing we miss is bucket refill rate.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.