Description

The go static file https://github.com/projectdiscovery/cdncheck/blob/dev/cidr_data.go should be refactored to use https://pkg.go.dev/embed, similarly to projectdiscovery/wappalyzergo#21

• JSON data should be written to https://github.com/projectdiscovery/cdncheck/blob/dev/cidr_data.json
• cidr_data.go should contain:

package cdncheck

import (
	_ "embed"
)

//go:embed cidr_data.json
var data string

• The data should be unmarshaled to InputCompiled{CDN: map[string][]string{}}}

This project started with base idea to detect if given IP/CIDR belongs to known CIDR ranges or not and further used with other ProjectDiscovery project to build additional feature on the top of it, for example, in naabu to avoid full port scanning if given IP is CDN, while having CDN information itself is useful information for the reconnaissance purpose but it might not be necessary that all the CDN should be excluded from the port scan, and there are also WAF which definitely needs to be excluded from the scan, for example: projectdiscovery/naabu#341, this can be easily extended to adding WAF ranges into CDN, but to avoid further confusion and extend the list more, it's required to differentiate between WAF (CDNs) and Generic CDNs, as a results, we need to create another function, similar to isCdn to detect isWaf which can be further utilized by other tools as needed.

Additional context: #7

This differentiation will allow to be provide more flexible data, where WAF ranges can be utilized to exclude from extended port scan, and generic CDN ranges can be used for information gathering / display purpose.

Please support the use of ASN numbers for identifying cloud assets.

It is possible to detect akamai or a cloud/waf/cdn even when the IP is not in the ASN or CIDR list. We could implement all of them as fallback methods.
I've been trying to develop a tool (https://github.com/brenocss/whatcdn) as a laboratory to validate some of these detection methods and it has worked.

• CNAME method #27

$ echo www.gap.com | dnsx -cname -resp
www.gap.com [www.gap.com.edgekey.net]

• PTR Method we can reuse code from cname detection

$ echo $(echo www.gap.com | dnsx -resp-only) | dnsx -ptr -resp
23.52.118.13 [a23-52-118-13.deploy.static.akamaitechnologies.com]

• HTTP response / wappalyzer method #28
• Nuclei method
  https://github.com/projectdiscovery/nuclei-templates/blob/master/technologies/tech-detect.yaml
  https://github.com/projectdiscovery/nuclei-templates/blob/d6636f9169920d3ccefc692bc1a6136e2deb9205/technologies/waf-detect.yaml
  https://github.com/projectdiscovery/nuclei-templates/blob/d6636f9169920d3ccefc692bc1a6136e2deb9205/dns/dns-waf-detect.yaml

Based in this fallback methods we could do something like https://github.com/trickest/resolvers

If a ip is cloud/cdn/waf enough times with cname/ptr detection, it may earn its place in the next cidr list, preventing unnecessary http/dns requests

Similar to #25 we could also add is_cloud similar to https://github.com/99designs/clouddetect

From: dev

cd cmd/generate-index
go build

./generate-index 
2022/09/08 15:55:30 [error] Could not process: could not get url https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20220725.json: no cidrs found for url

• failed fetch investigation
• failed to fetch should be a warning and ignored instead of process exit and not updating other data.

#40 (comment)

$ cat list.txt

hackerone.com
hackerone.com:443
https://hackerone.com
https://hackerone.com:443
104.16.100.52
104.16.100.52/32
104.16.100.52:443
https://104.16.100.52
https://104.16.100.52:443

$ cat list.txt | ./cdncheck -resp

hackerone.com [waf] [cloudflare]
hackerone.com:443 [waf] [cloudflare]
https://hackerone.com [waf] [cloudflare]
https://hackerone.com:443 [waf] [cloudflare]
104.16.100.52 [waf] [cloudflare]
104.16.100.52/32 [waf] [cloudflare]
104.16.100.52:443 [waf] [cloudflare]
https://104.16.100.52 [waf] [cloudflare]
https://104.16.100.52:443 [waf] [cloudflare]

echo https://docs.hackerone.com | ./cdncheck -json

{
  "timestamp": "2022-08-26T13:03:53.211756+05:30",
  "input": "https://docs.hackerone.com"
  "host": "docs.hackerone.com"
  "ip": "185.199.108.153",
  "cname": "hacker0x01.github.io",
  "cdn": true,
  "cdn_name": "github",
  "type": "cname"
}

$ go install github.com/brenocss/whatcdn@latest

$ cat list.txt | whatcdn -t 20 -json

StackPath can be used as source for IP CDN checking. Since there is no public available list of IP addresses / CIDR ranges, ip.info can be used with AS AS64259.

I would update by myself the json, but I'm on the Free plan and fetching ASN with APIs requires at least a Basic Plan.

• update example code (using as lib) + integration test
• add default values in match/filter

   -mcdn, -match-cdn string[]      match host with specific cdn provider (azure,cloudflare,cloudfront)
   -fcdn, -filter-cdn string[]      filter host with specific cdn provider (azure,cloudflare,cloudfront)

• add color to the output

• add banner

echo hackerone.com | ./cdncheck -resp
            __           __              __  
  _________/ /___  _____/ /_  ___  _____/ /__
 / ___/ __  / __ \/ ___/ __ \/ _ \/ ___/ //_/
/ /__/ /_/ / / / / /__/ / / /  __/ /__/ ,<   
\___/\__,_/_/ /_/\___/_/ /_/\___/\___/_/|_|  
                                       
                        projectdiscovery.io

[INF] Current cdncheck version 1.0.0 (latest)
hackerone.com [waf] [cloudflare]

• add self update / version options

   -up, -update  update cdncheck to latest version

• add no color option / silent option

   -nc, -no-color  disable colors in cli output
   -silent  only display results in output

Description

It would be useful to have a CLI utility that could be used to process a list of IP/CIDRs in addition to the existing cdn-server endpoint

• Stdin/Stdout support (Cidr expansion)
• Filtering ip parts of cdn
• Allow/Deny IP

➜  ~ ls ~/go/bin/cdncheck
/root/go/bin/cdncheck
➜  ~ cdncheck -v

            __           __              __
  _________/ /___  _____/ /_  ___  _____/ /__
 / ___/ __  / __ \/ ___/ __ \/ _ \/ ___/ //_/
/ /__/ /_/ / / / / /__/ / / /  __/ /__/ ,<
\___/\__,_/_/ /_/\___/_/ /_/\___/\___/_/|_|

                projectdiscovery.io

[INF] Current cdncheck version v1.0.9 (latest)
[INF] No results found.

➜  ~ echo 104.110.243.51| cdncheck -waf -cdn -cloud


            __           __              __
  _________/ /___  _____/ /_  ___  _____/ /__
 / ___/ __  / __ \/ ___/ __ \/ _ \/ ___/ //_/
/ /__/ /_/ / / / / /__/ / / /  __/ /__/ ,<
\___/\__,_/_/ /_/\___/_/ /_/\___/\___/_/|_|

                projectdiscovery.io

[INF] Current cdncheck version v1.0.9 (latest)
[INF] No results found.

➜  ~

however 104.110.243.51 => AS20940

should be detected as akamai

hi we need to know how to use it or build the release show us some examples like always u do thank you

Thanks for making this repo; it's very useful! My question is, since I often need to detect whether an IP belongs to a cloud provider, and not strictly a CDN, would you accept a PR which adds cloud providers such as those listed here? Or is the scope of this repo restricted to CDNs only?

After this change, you could use cdncheck to detect whether an IP belonged to a cloud provider, e.g. give it the IP of an s3 bucket and it would return "amazon". Curious about your thoughts.

Description

It's impossible to use the functionality without accessing the cache endpoint or querying all the known CDN endpoints. It should be considered once a week, dropping the list into a static go file via GitHub action, exactly as it's done with https://github.com/projectdiscovery/wappalyzergo. This will speed up processing and reduce the amount of network traffic.

• The cdn_check function should be able to receive a DNS response as parameter, and use it to define if it is a CDN or not.
  In this way, we can extend the detection methods, such as checking the cnames to define if it is a CDN.
• Cdn check receiving a dns response as a parameter will prevent us from doing new unnecessary dns resolutions. For example in dnsx https://github.com/projectdiscovery/dnsx/blob/a1e4472b4736a2f2350d00ab2410ffeaac3381d4/libs/dnsx/cdn.go#L11 we do a new dns resolution, and we are skipping resolvers

func check_cdn_with_dns(dnsResponse *retryabledns.DNSData) (bool, string, string) {

	iscdn_cname, cdn_name, detection_type := check_cdn_cname_list(dnsResponse.CNAME)
	if iscdn_cname {
		return true, cdn_name, detection_type
	}
	// if in_ipv6_list(dnsResponse.AAAA) {
	// 	return true
	// }
	iscdn_ip, cdn_name, detection_type := ip_list_cdn_check(dnsResponse.A)
	if iscdn_ip {
		return true, cdn_name, detection_type
	}
	return false, "", ""
}

Using cname to detect if it belongs to a CDN. We could also do it to detect if has a waf https://github.com/projectdiscovery/nuclei-templates/blob/master/dns/dns-waf-detect.yaml #25

func check_cdn_cname_list(cname_list []string) (bool, string, string) {
	CDN_DOMAINS := map[string]string{
		"cloudfront.net":         "amazon",
		"amazonaws.com":          "amazon",
		"edgekey.net":            "akamai",
		"akamaiedge.net":         "akamai",
		"akamaitechnologies.com": "akamai",
		"akamaihd.net":           "akamai",
		"cloudflare.com":         "cloudflare",
		"fastly.net":             "fastly",
		"edgecastcdn.net":        "edgecast",
		"impervadns.net":         "incapsula",
	}

	for _, cname := range cname_list {
		// check if this cname ends with a CDN domain
		for cdn_domain, cdn_name := range CDN_DOMAINS {
			if strings.HasSuffix(cname, cdn_domain) {
				return true, cdn_name, "CNAME"
			}
		}
	}
	return false, "", ""
}

When we detect a ip that is not detected by ip list, we should add to the ip_list.

Projects that are used as both CLI and library are affected by this; automated weekly releases for libraries cause issues as it does not account for -

• automatic banner version update
• the automated tag creation (github-actions) doesn't trigger the binary release workflow.
  • reference - https://github.com/orgs/community/discussions/27028

The same issues apply to MapCIDR project as well.

Note: The weekly automatic tag creation workflow is disabled.

Continuation of #37

• CLI flag add / update
• JSON output

CLI option:

cdncheck is a tool for identifying the technology associated with ip network addresses.

Usage:
  ./cdncheck [flags]

Flags:
INPUT:
   -i, -inputs string[]  input ip / cidr to process
   -l, -list string      file input with list of ip / cidr to process

DETECTION:
   -cdn              display cdn ip
   -cloud            display cloud ip
   -waf              display waf ip

MATCHERS:
   -mcdn, -match-cdn string[]         match host with specified cdn provider (fastly, incapsula)
   -mcloud, -match-cloud string[]     match host with specified cloud provider (fastly, incapsula)
   -mwaf, -match-waf string[]         match host with specified waf provider (fastly, incapsula)

FILTER:
   -fcdn, -filter-cdn string[]        filter host with specified cdn provider (fastly, incapsula)
   -mcloud, -filter-cdn string[]      filter host with specified cloud provider (fastly, incapsula)
   -mwaf, -filter-cdn string[]        filter host with specified waf provider (fastly, incapsula)
   -e, -exclude      		      exclude detected ip from output

OUTPUT:
   -resp             display technology name in cli output
   -j, -json         write output in json format
   -o, -output       write output in plain format
   -version          display version of the project

JSON output:

{
  "timestamp": "2022-08-02T03:00:17.698783+05:30",
  "ip": "104.16.51.111",
  "cdn": true,
  "cdn_name": "cloudflare"
}

CLI examples:

$ cat ips

52.60.165.183
3.98.63.202
52.60.160.16
185.199.109.153
104.16.51.111
54.192.171.16
54.192.171.2
54.192.171.22

$ cat ips | cdncheck

104.16.51.111 
54.192.171.16
54.192.171.2
54.192.171.22

$ cat ips | cdncheck -resp

104.16.51.111 [waf] [cloudflare] 
54.192.171.16 [waf] [cloudflare] 
54.192.171.2 [cdn] [cloudfront] 
54.192.171.22 [cdn] [cloudfront] 

$ cat ips | cdncheck -resp -waf

104.16.51.111 [waf] [cloudflare] 
54.192.171.16 [waf] [cloudflare]

$ cat ips | cdncheck -e

52.60.165.183
3.98.63.202
52.60.160.16
185.199.109.153

$ cat ips | cdncheck -e -waf

52.60.165.183
3.98.63.202
52.60.160.16
185.199.109.153
54.192.171.2
54.192.171.22

Question: -e should already exclude -waf?

https://github.com/projectdiscovery/cdncheck/blob/dev/other.go#L22

Would be great to include CheckSuffix function ( https://github.com/projectdiscovery/cdncheck/blob/dev/other.go#L22 ) and CheckWappalyzer as fallback as well

Originally posted by @brenocss in #67 (comment)

Even though this PR appears to add wappalyzer support to cdncheck, I'm finding that not to be true.

Example

For example - store-ca.figma.com

This site uses Cloudflare as it's CDN but is NOT hosted on a Cloudflare IP. This appears to be because Shopify (the host) must be using the Cloudflare BYOIP (bring your own IP) program.

cdncheck -i store-ca.figma.com
Returns "No results found"

The IP address (23.227.38.74) is owned by Shopify themselves

However
nuclei -u https://store-ca.figma.com -id tech-detect

Returns cloudflare as the HTTP header shows it is clearly being served via Cloudflare CDN

Code

In looking at the code (other.go) there is a function CheckWappalyzer .. but that code isn't called anywhere except in other_test.go.

This means when running naabu against this same target, the -exclude-cdn flag doesn't correctly identify this target as served by a CDN

Description

Only ipv4 cidrs are supported via regex extraction

To allow user to extend the known CDN lists by providing known ASN or CIDR with name, a configurable YAMl file can be used to maintain the list, for example:

cdn:
  asn:
    akamai:
    - AS12222

    sucuri:
    - AS12222

    leaseweb:
    - AS12222

  cidr:
    fastly:
    - 23.235.32.0/20
    - 43.249.72.0/22
    - 103.244.50.0/24
    - 103.245.222.0/23
    - 103.245.224.0/24
    - 104.156.80.0/20
    - 140.248.64.0/18
    - 140.248.128.0/17
    - 146.75.0.0/17
    - 151.101.0.0/16
    - 157.52.64.0/18
    - 167.82.0.0/17
    - 167.82.128.0/20
    - 167.82.160.0/20
    - 167.82.224.0/20
    - 172.111.64.0/18
    - 185.31.16.0/22
    - 199.27.72.0/21
    - 199.232.0.0/16

Currently it's hardcodd here - https://github.com/projectdiscovery/cdncheck/blob/master/ranges.go, with this change, information from YAML file be will be used to populate CIDR ranges based on ASN input for the API service.

CDN detection for IP using PTR DNS records as fallback.

I believe this feature can work, but I think it's worth a more in-depth analysis.

$ host -t ptr 104.87.29.118

118.29.87.104.in-addr.arpa domain name pointer a104-87-29-118.deploy.static.akamaitechnologies.com.
ns7.reverse.deploy.akamaitechnologies.com has address 184.26.161.32
ns5.reverse.deploy.akamaitechnologies.com has address 95.100.173.32
ns1.reverse.deploy.akamaitechnologies.com has address 96.7.50.32
ns3.reverse.deploy.akamaitechnologies.com has address 23.74.25.32
ns8.reverse.deploy.akamaitechnologies.com has address 2.22.230.32
ns6.reverse.deploy.akamaitechnologies.com has address 23.61.199.32
ns2.reverse.deploy.akamaitechnologies.com has address 2.16.40.32
ns4.reverse.deploy.akamaitechnologies.com has address 95.100.168.32

to add this feature we can add

	if dnsResponse.PTR != nil {
		matched, discovered, itemType, err := c.CheckSuffix(dnsResponse.PTR...)

check_cdn should also receive, as a parameter, the struct that wappalyzer returns. This will allow, for example, that in httpx the result of tech_detect interferes with the response of is_cdn. Therefore, if an ip is not in the cdn list, but is a cdn detected via wappalyzer, add this ip to the cdn ip list.

func check_cdn_with_wappalyzerResponse(wappalyzerResponse map[string]struct{}) (bool, string, string) {
	detection_type := "HTTP"
	CDN_TECHNOLOGIES := map[string]string{
		"imperva":    "incapsula",
		"incapsula":  "incapsula",
		"cloudflare": "cloudflare",
		"cloudfront": "amazon",
		"akamai":     "akamai",
	}

	// matches := wappalyzerClient.Fingerprint(httpResponse.Header, body)

	for technology := range wappalyzerResponse {
		for cdn_tech, cdn_name := range CDN_TECHNOLOGIES {
			// check if technology.lower contains a CDN technology
			if strings.Contains(strings.ToLower(technology), cdn_tech) {
				return true, cdn_name, detection_type
			}
		}
	}
	return false, "", detection_type
}

check_cdn fails to detect the www.gap.com ip as an akamai ip. However, wappalyzer was able to detect the akamai technology in the http response. In this way, httpx should return as a result that belongs to a cdn, return the name of the cdn, skip port scans (exclude cdn) and add this ip to the list of cdn ips of project discovery

echo https://www.gap.com | httpx -td -json

this would be a nice addition to #25 as well

While I was doing my PR #13, I was testing whether the CIDR was really scraped or not.
Apparently, there are several providers that do not produce CIDRs, those are Akamai & Sucuri.

Then I checked where the CIDR resource for that provider is hit the endpoint, and it comes from ipinfo.io. Then I tried to hit via cURL, but it resulted in an error with a message: required access token.

The conclusion is, to get the CIDR resource from ipinfo.io an access token is required. Are there any other resources to get the CIDR range for those provider? Or it just me who missed it? But I'm not sure, since the scrape... function itself doesn't returning an error if CIDR slice is less than 1.

I am creating this issue mainly to share a resource.

This may be a useful addition. It contains 10's of common CDN domains.
https://github.com/WPO-Foundation/wptagent/blob/master/internal/optimization_checks.py#L67-L328

other tools, don't call the pdtm-api when --version flag is used. make it same for cdncheck

Hi,

You can also add Google CDN IP ranges which are published on their officially updated endpoint:

https://www.gstatic.com/ipranges/cloud.json

Regards.

❯ cdncheck -update

            __           __              __  
  _________/ /___  _____/ /_  ___  _____/ /__
 / ___/ __  / __ \/ ___/ __ \/ _ \/ ___/ //_/
/ /__/ /_/ / / / / /__/ / / /  __/ /__/ ,<   
\___/\__,_/_/ /_/\___/_/ /_/\___/\___/_/|_|

		projectdiscovery.io

[updater] executable cdncheck not found in release asset `0` got: update: could not find release asset for your platform (linux/amd64)

It is happening because, cdncheck latest release does not include the assets, it is released by github action bot

cd cmd/generate-index
go build .
./generate-index

2022/10/19 17:14:39 [err] could not fetch cloud item: could not get url https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20220905.json: no cidrs found for url

The issue (#46) looked fixed in #47, but the new URL still gives 404 as a response.

• As this issue #27 suggests, we could change cdncheck behavior to handle dns_resposes. therefore, I suggest this workflow if we just have dns_response

1. handle dns_response
2. get ip's from dns_response
3. check if belongs to the project discovery cdn ip list ( if does return true , cdn_name )
4. if not, get cnames from dns_response
5. check if belongs to cname cdn list ( if does return true , cdn_name and add this ip to cdn ip list )

• As this issue #28 suggests , we can also handle the tech_detection response to determine if it belongs to a cdn/waf. we could add this scan as well. I suggest this workflow if we have dns_response + http_response.

1. handle dns_response
2. get ip's from dns_response
3. check if belongs to the project discovery cdn ip list ( if does return true , cdn_name )
4. if not, get cnames from dns_response
5. check if belongs to cname cdn list ( if does return true , cdn_name and add this ip to cdn ip list )
6. if not, check http response technologies.
7. check if any technology belongs to any cdn/waf ( if does return true , cdn_name and add this ip to cdn ip list )