no results if testing cloudflare protected urls about subbrute HOT 5 CLOSED

@jeteon do you have repro steps?

Testing waveapps.com I'm seeing stuff like this returned (names_temp.txt used below is just a text file with 2 entries: www and enlfneakfeanlfafn, the second being a long one I know can't exist):

root@kali-vm-212:/opt/subbrute# python subbrute.py -p waveapps.com -s names_temp.txt
waveapps.com,13,1a506c656173652073746f702061736b696e6720666f7220414e591f5365652064726166742d696574662d646e736f702d7265667573652d616e79
www.waveapps.com,13,1a506c656173652073746f702061736b696e6720666f7220414e591f5365652064726166742d696574662d646e736f702d7265667573652d616e79
enlfneakfeanlfafn.waveapps.com,13,1a506c656173652073746f702061736b696e6720666f7220414e591f5365652064726166742d696574662d646e736f702d7265667573652d616e79

If you look at the results, it is not actually saying those domains exist. This is an oddity related to Cloudflare when requesting record type ANY, which I believe subbrute does. The result returned above is in the format:
domain name, dns record type, record

So in the above results, the record type is 13 (HINFO), not a valid response containing an IP.

I haven't dug in to subbrute enough to know why the record looks like a long hash, but if you use dig and attempt an ANY type record lookup against that host you can see the response Cloudflare puts in for an ANY request: Please stop asking for ANY:

root@kali-vm-212:/opt/subbrute# dig aaaaaaa.waveapps.com any

; <<>> DiG 9.10.3-P4-Debian <<>> aaaaaaa.waveapps.com any
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43257
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; MBZ: 0005 , udp: 1280
;; QUESTION SECTION:
;aaaaaaa.waveapps.com.		IN	ANY

;; ANSWER SECTION:
aaaaaaa.waveapps.com.	5	IN	HINFO	"Please stop asking for ANY" "See draft-ietf-dnsop-refuse-any"

;; Query time: 55 msec
;; SERVER: 192.168.121.2#53(192.168.121.2)
;; WHEN: Wed Jan 18 17:35:00 EST 2017
;; MSG SIZE  rcvd: 120

So, with Cloudflare IPs I think there are a few possible solutions:

1. Ignore HINFO (13) type responses by logging to a file, then grepping with -v ",13,"
2. Change line 550 of subbrute.py to suppress printing by changing it from:

        if result:

to:

        if result and record_type != '13':

3. Get fancy by adding a command line flag (--nohinfo or similar) to ignore these types of responses, then submit a PR so I can use it too :)

For more reading on why Cloudflare does this, they made a blog post here: https://blog.cloudflare.com/what-happened-next-the-deprecation-of-any/

I hope that helps clear things up,
dg

from subbrute.

use proxychains and multiple proxies that come from trusted sources? even then you might trigger CAPTCHAs.. there's nothing this tool can really do about that.

from subbrute.

subbrute works with cloudflare. If you are having trouble with a specific domain, then please provide reproduction steps.

from subbrute.

I've also come across issues when the domains use Cloudflare for their nameservers. For instance, if you try to brute force the sub-domains of waveapps.com or ramnode.com then every sub-domain tried will come back as existing, even though it doesn't. Others that seem to have a slightly different setup but also use Cloudflare nameservers like kraken.com seem to work as expected though. I suspect the nameserver might behave differently based on what plan you're on but can't be sure yet. Looking further into it though.

from subbrute.

Thanks for the pointer. It seems that some of these Cloudflare domains will actually return the HINFO(13) response regardless of whether the domain exists or not which is I guess why the application struggles with brute-forcing these domains. I tried the method of blacklisting the HINFO(13) responses as you suggested but that resulted in everything being blacklisted because everything returns HINFO(13) but it certainly got me thinking. The link you provided was extremely helpful and seem to indicate that they gradually introduced this.

I think the fix here would be to detect this when resolving a sub-domain known to exist and then use the ALL instead of the ANY in these cases. Its a bit tricky for now since you'd have to know upfront that the authoritative name server is doing this. For now, I guess if you know up front that the domain you are testing has this issue you have to remember to use --type=ALL.

from subbrute.