honoki / bbrf-client Goto Github PK

Hello! Thanks for the awesome tool, you guys have done a great job so far!

I think it would be great if one can query all urls with query parameters so its possible to feed them to another tool (think dalfox / sqlmap). At the moment the parameters are not returned when querying all urls:
bbrf urls

Since its storing the parameter values in array:
bbrf show https://hostwithqueryparams.xyz

{ ... "hostname": "hostwithqueryparams.xyz", "port": 443, "status": 200, "content_length": 298820, "query": [ "cat=1&catb=136", "product=4847" ] ...

The syntax could be:
bbrf urls -params

Which returns all urls with query parameters:
https://hostwithqueryparams.xyz?cat=1&catb=136
https://hostwithqueryparams.xyz?product=4847
...

Thanks! :)

Sometimes I would like to know the program from which a domain is coming from, for example, somedomain.com

bbrf search domain somedomain.com

Output

> _Program

Search IP

bbrf search ip 1.1.1.1

Output

> Program0 Program1 ... Program n

Search URL

bbrf search http://www.test.com

Output
> Program_0

It might be something like:

bbrf urls --enabled or that by default bbrf url with print only the urls from enabled programs and using --all will retrieve them all. Same with domains and ips.

I'm trying to add a big program, around 5.5 million domains, the inscope and outscope are big also.

I don't know why it's failing, this is what I tried:

cat domains-big.txt | bbrf domain add - -s subfinder --show-new

Doesn't add any domain nor output any error.

If I do this sed -n '50001 ,100000p' domains-big.txt| bbrf domain add - -s subfinder --show-new

I get this error "[ERROR] local variable 'success' referenced before assignment"

The same error doing:

awk 'NR > 50000 && NR <= 100000' domains-big.txt | bbrf domain add - -s subfinder --show-new

Any idea why this might happen?

The error handling at the moment is not very clear, as the client just dumps the error to stdout. Ideally, the error message is concise and shows actionable information that will help the end user solve any potential issues.

when trying to delete a blacklist element from a program, I get bellow error

$ bbrf blacklist add www.example.com -p example.com
$ bbrf blacklist remove www.example.com -p example.com
Traceback (most recent call last):
  File "/root/bbrf-client/bbrf.py", line 935, in <module>
    result = bbrf.run()
  File "/root/bbrf-client/bbrf.py", line 884, in run
    self.remove_blacklist(self.arguments['<element>'])
  File "/root/bbrf-client/bbrf.py", line 694, in remove_blacklist
    blacklist.delete(e)
AttributeError: 'list' object has no attribute 'delete'

Hi @honoki,

A collaborator created a program with name 'AT&T' (using bbrf new "AT&T"). The program was created correctly and we could add domains, but other functions failed:

bbrf use "AT&T" Worked

> bbrf domains
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1)
DEBUG:urllib3.connectionpool: "GET /bbrf/_design/bbrf/_view/domains?reduce=false&key=%22AT&T%22 HTTP/1.1" 400 54
[ERROR] 'rows'

> bbrf urls
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1)
DEBUG:urllib3.connectionpool: "GET /bbrf/_design/bbrf/_view/urls_by_program?reduce=false&key=%22AT&T%22 HTTP/1.1" 400 54
[ERROR] 'rows'

It would be easy for me to recreate the program without the symbol, I just wanted to point it out.

I have tested this situation many times:

1. Running a heavy process (deleting over 100k urls or domains)
2. Press Control + C
3. Processes on the CouchDB server keep running. Sometimes overloading the server.

I often need to restart the DB in order to get rid of the left over processes.
This might be the cause for the 'unknown error' described in another issue.

Ability to search db by keywords
example:

> bbrf domains -p vzm -s omega
something.omega.gq1.yahoo.com
omega.gq1.yahoo.com
gq1.something.omega.yahoo.com

Sometimes I face errors like this one

Hostname could not be parsed, skipping http://

I have no idea what the hostname is or what might have cause the error.

I would be useful to increase the verbose level to show what hostname was giving troubles.

I was running cat domains.txt |bbrf domain add - -s subfinder --show-new and got this error:

[ERROR] string indices must be integers

I found some domains (from subfinder) starting with '_', character that is not allowed.

It would be cool/nice to have a warning/error message when these type of formatting errors are found.

• Try adding any out of scope domain
• No error message is thown and nothing gets added as well

I was using the option "debug": "true" changed it to "debug": "false" but still shows the debug message.

Thanks.

Sometimes I need to update or modify a program tag, ie:

bbrf new programX

bbrf new programX -update -t hackerone

or

bbrf update programX -t hackerone

Hello there,

Managed to install CouchDB and the client, I needed to modify the file bbrf_api.py to allow requests to my server with a self-signed certificate, so I just added verify=False to all self.requests_session functions.

I manage to create programs and their scope/domains, but I can't retrieve them:

 > bbrf programs 

Traceback (most recent call last):
  File "/home/kali/software/bbrf-client/bbrf.py", line 828, in <module>
    result = bbrf.run()
  File "/home/kali/software/bbrf-client/bbrf.py", line 666, in run
    return self.list_domains(self.arguments['--all'])
  File "/home/kali/software/bbrf-client/bbrf.py", line 565, in list_domains
    return self.api.get_domains_by_program_name()
  File "/home/kali/software/bbrf-client/bbrf_api.py", line 45, in get_domains_by_program_name
    raise Exception(r.json()['error'])
Exception: not_found

> bbrf domains --all

Traceback (most recent call last):
  File "/home/kali/software/bbrf-client/bbrf.py", line 828, in <module>
    result = bbrf.run()
  File "/home/kali/software/bbrf-client/bbrf.py", line 666, in run
    return self.list_domains(self.arguments['--all'])
  File "/home/kali/software/bbrf-client/bbrf.py", line 565, in list_domains
    return self.api.get_domains_by_program_name()
  File "/home/kali/software/bbrf-client/bbrf_api.py", line 45, in get_domains_by_program_name
    raise Exception(r.json()['error'])
Exception: not_found

bbrf new OK 
/home/kali/software/bbrf-client/.env/lib/python3.9/site-packages/urllib3/connectionpool.py:981: InsecureRequestWarning: Unverified HTTPS request is being made to host 'info-sec.cl'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(

What could be the problem?

Hello @honoki,
I have the following program:

> bbrf show program
{"_id":"domain","_rev":"4-d573602c57707b62902e6ba77993d56f","type":"program","disabled":false,
"passive_only":false,"inscope":"*.domain.com","*.domainmoney.com","*.domainmall.com",
"*.domainbank.com","domain.com","outscope":[],"tags":{"site":"self","reward":"money"}}

I was trying to query programs by tag in the following way: bbrf programs where site is self but I have no results.

Same happens with tag reward bbrf programs where reward is money I also tried using " and ' in tags names and key values.

At the moment, you can only add domains and URLs within a program (i.e. with he -p flag, or in the currently active program). It might be useful to have a "bulk add" feature that takes the input and adds it to any programs that match the input, regardless of the specified or active program. For example, this could look like:

cat urls_all.txt | bbrf url add - --cross-program
cat domains_all.txt | bbrf domain add - --cross-program

BBRF does not yet support services, which would be a nice addition to urls, ips and urls. The idea would be to store the results of network scans like nmap or masscan that discovered open services.

Inspiration for some supported attributes for services can be gathered from this nmap-to-mongo project:

• ip
• port
• state
• service
• hostname
• ostype
• ...

A useful unique key (_id) would likely be the concatenation of ip:port so that it can feed back into other scanners, e.g. like this:

# retrieve all known services (i.e. known open ports) and store URLs when the service is a valid HTTP service
bbrf services | httpx | bbrf url add - -t root:true

Thanks to @pry0cc for the valuable input!

I was adding some domains and I had this error:

[ERROR] too many values to unpack (expected 2)

I discovered it was because of a IPv6 (therefore the : symbol)

Doing this solved the issue:

cat domains.txt | grep -v ":" | bbrf domain add - -s subfinder --show-new

By supporting custom properties to be set for documents, you could specify e.g. the platform a program belongs to or the name of a team member that added a bunch of new domains.

This could look as follows:

bbrf new example --set platform:intigriti
bbrf domain add 'www.example.com' --set added_by:pieter 'comment:just stumbled across this'

This probably requires some checks on property names to ensure the integrity of the information, e.g. no reserved keywords like id, _id, _rev or bbrf-related fields like domains, ips, url, type, etc. should be allowed.

An example use case where this would be useful is if the outscope changes, and a number of domains in the database "become outscoped". BBRF does not retroactively remove those domains from the database (nor should it), but it would be nice to do something like bbrf domains | bbrf scope filter out to display the list of domains in the database that match the outscope.

As a result, it becomes easy to remove outscoped domains from the database:

bbrf domains | bbrf scope filter out | bbrf domain remove -

In general, the flags --all --show-disabled are meant to "return data from all programs, including disabled programs". For some reason, it seems I originally implemented bbrf scope in --all --show-disabled to mean "get scope of ONLY disabled programs.

This is inconsistent, which means I'll likely change this to either use a new flag --only-disabled, or to return scope of both enabled and disabled programs.

Instead, ensure you use bbrf programs to list all programs, compatible with the where syntax to query custom tags on programs:

# add tags when creating a program
bbrf new example -t priority:high -t platform:hackerone

# filter programs on custom tags
bbrf programs where platform is hackerone
bbrf programs where priority is high

# update program tags with a new command
bbrf program update example -t priority:low
# or remove tags as follows
bbrf program update example -t priority:
# or update multiple programs in-line:
bbrf program update example example2 -t custom:tag
# to add a custom tag to all programs at once:
bbrf programs | bbrf program update - -t custom:tag

Hi, it would be great if we can just use the bbrf domain add asdf.example.com and bbrf add it to the matching program without the need to use the -p option

Try adding an out of scope domain.
in line https://github.com/honoki/bbrf-client/blob/master/bbrf.py#L246
add_documents gets called with empty add_domain object. Wasted call.

It takes more than a minute to execute:

> bbrf urls -p PROGRAM where source is 'httpx'

Debug command (~ 326K urls):

> time curl $(jq -r .couchdb ~/.bbrf/config.json)'/_design/bbrf/_view/search_tags?key=\["source","httpx"\]' -i -u bbrf:password


 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100 22.7M    0 22.7M    0     0  3892k      0 --:--:--  0:00:05 --:--:-- 3883k

real	0m6.034s
user	0m0.315s
sys	0m0.260s

Hi,
I get bellow error when I try to add a domain to a program with the tag option -t:

# bbrf domain add www.example.com -p example.com -t test
Traceback (most recent call last):
  File "/root/bbrf-client/bbrf.py", line 935, in <module>
    result = bbrf.run()
  File "/root/bbrf-client/bbrf.py", line 765, in run
    return self.add_domains(self.arguments['<domain>'])
  File "/root/bbrf-client/bbrf.py", line 255, in add_domains
    success, _ = self.api.add_documents('domain', add_domains, self.get_program(), source=self.arguments['-s'], tags=self.arguments['-t'])
  File "/root/bbrf-client/bbrf_api.py", line 233, in add_documents
    tag_map = {x.split(':', 1)[0]: x.split(':', 1)[1] for x in tags}
  File "/root/bbrf-client/bbrf_api.py", line 233, in <dictcomp>
    tag_map = {x.split(':', 1)[0]: x.split(':', 1)[1] for x in tags}
IndexError: list index out of range

Commands

bbrf new 1testing -t site

Traceback (most recent call last):
  File "/home/kali/.local/bin/bbrf", line 8, in <module>
    sys.exit(main())
  File "/home/kali/.local/lib/python3.9/site-packages/bbrf/bbrf.py", line 986, in main
    result = bbrf.run()
  File "/home/kali/.local/lib/python3.9/site-packages/bbrf/bbrf.py", line 771, in run
    self.new_program()
  File "/home/kali/.local/lib/python3.9/site-packages/bbrf/bbrf.py", line 101, in new_program
    self.api.create_new_program(self.get_program(), tags=self.arguments['-t'])
  File "/home/kali/.local/lib/python3.9/site-packages/bbrf/bbrf_api.py", line 39, in create_new_program
    tag_map = {x.split(':', 1)[0]: x.split(':', 1)[1] for x in tags}
  File "/home/kali/.local/lib/python3.9/site-packages/bbrf/bbrf_api.py", line 39, in <dictcomp>
    tag_map = {x.split(':', 1)[0]: x.split(':', 1)[1] for x in tags}
IndexError: list index out of range

And then without the -t flag

> bbrf new 1testing
(no output) 

I was adding some urls after defining a the scopes and some were rejected due to the domain name not being in scope.
Then I realized the scope was not updated because it contains upper case letters.

> bbrf inscope add bugcrowdcontentAPI.site.cloud
[DEBUG] getting program scope
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): XX:6984
DEBUG:urllib3.connectionpool:XX:6984 "GET /bbrf/Softdocs HTTP/1.1" 200 176

> bbrf scope in
[DEBUG] getting program scope
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): XX:6984
DEBUG:urllib3.connectionpool:https://XX:6984 "GET /bbrf/Softdocs HTTP/1.1" 200 176

Then using api instead of API

> bbrf inscope add bugcrowdcontentapi.site.cloud
[DEBUG] getting program scope
truncated ...
[DEBUG] updating program scope
truncated ...

> bbrf scope in
[DEBUG] getting program scope
truncated ...
bugcrowdcontentapi.site.cloud

I guess case sensitivity in domain names should be ignored or warned.

When you update an nonexistent program BBRF gives an empty response.

In debug mode, when the program exists displays POST /bbrf/_bulk_docs HTTP/1.1

It would be useful to have an error message for when you enter a typo and don't check the updated info.

I was testing the use of this syntax to retrieve urls by source:

> bbrf urls -p PROGRAM where source is 'httpx'

It takes some time (more than 1 minute, but it works)

But, if I do the following:

bbrf use PROGRAM

bbrf urls where source is 'httpx' 

It retrieves all urls (from all programs with the source 'httpx')

I'm using v1.1.7 and the latest server update.

I can add program, inscope, out scope, but I can't add domain to the program.
> bbrf show starbucks
{"_id":"starbucks","_rev":"2-6e239c74c5dc4d17055ab4cbe087c18d","type":"program","disabled":false,"passive_only":false,"inscope":["www.starbucks.com"],"outscope":[]}

When I type :
> bbrf domain add 1.starbucks.com -p starbucks
and then:
> bbrf domains
its output is none.

Hello there @honoki

In Hackerone some programs have a Burp Suite Project Configuration file (example
shopify.txt) with the scope and outscope assests. It would be great to be able to import that file to bbrf.

Hi @honoki,

I'm not sure if this feature maybe in your dev plans. So far I haven't found a way to query the data by source value.
I'm expecting something like this:

> bbrf urls where source is 'httpx' 

OR

> bbrf urls -s 'httpx' 

if sub list contains colons bbrf would consider it resolved and will add it to resolved list

Hi,

Really loving this tool, one feature I'd like to see is the ability to use Discord instead of Slack for alerts. This way people collaborating on Discord can have a more streamlined workflow instead of having to switch between applications.

Preferably this would be done with webhooks.

Thanks.

python3 bbrf.py 
Traceback (most recent call last):
  File "/home/kali/software/bbrf-client/bbrf/bbrf.py", line 48, in <module>
    from . import bbrf_api
ImportError: attempted relative import with no known parent package

After a fresh install. A workaround is removing "from ."

A common task for me is to discover where a subdomain/url is coming from (program + platform) it would be super useful to have something like this:

> bbrf show domain.example -t 

{"_id":"domain.example","_rev":"2-f0XXXX4226734cf9da7002e6","ips":["91.199.XX.XX"],"program":"Program","type":"domain","source":"subfinder", "tags": "key:value" }

Thanks!

When starting with a new program and defining scope only with:

bbrf scope in add *.example.com

trying to add the domain example.com will fail.

For replying the behaviour try:

bbrf new example
bbrf use example
bbrf inscope add *.example.com
bbrf domain add example.com
bbrf domains

Output will be empty.

Since the BBRF server stores services with their ip:port identifier, adding the same service when it is already assigned to another program will fail.

In practice, this might occur for either local IPs (although unlikely to be the result of a portscan), as well as when scanning shared infrastructure.

An improvement would be to either allow assigning more than one program to a single service, e.g. "program": ["one", "two"] rather than a single value "program": "one", and improving the server so it returns the document regardless of which program is specified.

I don't see the option on the bbrf client.

Hi, I usually create programs with the parent org's domain name for example

bbrf new yahoo.com
bbrf inscope add 'yahoo.com' -p yahoo.com
bbrf domain add yahoo.com -p yahoo.com

The last step always fails without error output in debug mode, is there a way around it, other than to rename all my program names? Thanks!

debug output:

┌──(root💀kali)-[~]
└─# bbrf domain add yahoo.com -p yahoo.com                                                                                                                                                   1 ⨯
[DEBUG] getting program scope
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): IP:PORT
DEBUG:urllib3.connectionpool:http://IP:PORT "GET /bbrf/yahoo.com HTTP/1.1" 200 3167
[DEBUG] getting program blacklist
[DEBUG] adding documents in bulk
DEBUG:urllib3.connectionpool:http://IP:PORT "POST /bbrf/_bulk_docs HTTP/1.1" 201 77

┌──(root💀kali)-[~]
└─# bbrf domains -p yahoo.com|egrep '^yahoo.com'
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): IP:PORT
DEBUG:urllib3.connectionpool:http://IP:PORT "GET /bbrf/_design/bbrf/_view/domains?reduce=false&key=%22yahoo.com%22 HTTP/1.1" 200 None

There is currently a known issue when configuring multiple programs with overlapping scope to a single bbrf instance, when trying to add a domain to each of the programs.

For example, when program1 and program2 both have *.example.com in their inscope, the command bbrf domain add www.example.com -p program1 will add a new document with identifier www.example.com to the database, and a subsequent bbrf domain add www.example.com -p program2 will silently fail without adding the new document, because a document with the same key already exists.

As a result, listing domains for program 2 with bbrf domains -p program2 will not include www.example.com.

A possible solution would be to store program names as part of the document key, but this introduces a number of other disadvantages. Another solution might be to issue a warning when specifying a scope that overlaps with the scope of another program.

Hi! Thanks for the new release! Really great job, again!

To the point... I don't know if I have this problem locally or not, but when trying to query all urls with certain port number with command like:
bbrf urls where port is 443, the command does not result any urls despite the fact that I most certainly have urls with port 443:
bbrf show https://test.xyz
..."hostname":"test.xyz","port":443, ...

However when trying to query with source or custom tag(s), it works flawlessly. Thanks again :)

First off, thanks so much for this awesome tool. Got txt files everywhere and this is a total lifesaver. Just started using it after watching you on Nahamcon.

At the moment URLs support status code and content length but it would be amazing to have the ability to pipe domains into httpx and store the site title, webserver and content type too. That way you can just query for any url that had XYZ in the title or Apache in the webserver.

Example httpx output:
http://sub.domain.com:80 [200] [178] [text/html] [I'm the title] [nginx]

Example command:
bbrf urls where title is "I'm the title"
bbrf urls where webserver is "nginx"

Again, thanks a million for this tool.

I ran one of the examples and this is my output:

 bbrf new vzm
 bbrf inscope add '*.yahoo.com' '*.yahoo.be'
Traceback (most recent call last):
  File "/home/ubuntu/bbrf-client/bbrf.py", line 753, in <module>
    result = bbrf.run()
  File "/home/ubuntu/bbrf-client/bbrf.py", line 667, in run
    self.add_inscope(self.arguments['<element>'])
  File "/home/ubuntu/bbrf-client/bbrf.py", line 483, in add_inscope
    (inscope, outscope) = self.api.get_program_scope(self.get_program())
  File "/home/ubuntu/bbrf-client/bbrf.py", line 126, in get_program
    raise Exception('You need to select a program to execute this action.')
Exception: You need to select a program to execute this action.

Hello,

I've been using bbrf and i have the following situation:

• *.example.com is in scope
• www.example.com is out of scope

What do will bbrf do in this particular case? is outscope "stronger" than in? In my particular case i filled inscope of several programs automatically, then i checked for example.com program and filled out of scope after adding www.example.com as a valid domain and url. Then when added www.example.com to out of scope, bbrf kept showing www.example.com when using bbrf urls.

If this is intended behaviour, could bbrf outscope add remove both domains and urls?

Thanks in advance

IP resolutions are not correctly parsed and stored when passing multiple <domain>:<ip> records for a single IP, which leads to innacuracies in the dataset.

E.g.:

bbrf domain add test.example.com:1.1.1.1 test.example.com:2.2.2.2
bbrf show test.example.com | jq
{
  "_id": "test.example.com",
  "_rev": "1-83edf0d0046f8dd062845c996aa8488f",
  "ips": [
    "2.2.2.2"
  ],
  "type": "domain",
  "program": "example"
}

Expected would be that both resolutions of the domain are stored.

Currently we can do bbrf program scope --wildcard to fetch the domains in scope for the active program, but unless I've missed something there isn't a way to fetch all domains in scope for all programs. This makes automation difficult without looping over each domain and setting it with bbrf use.

Hi Honoki,

I was trying to install fresh copy of bbrf-client using new method using pip3 install bbrf (my VPS has both python2 and python3 installed).

But after installation I'm getting this error:

$ pip3 install bbrf
Collecting bbrf
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/b6/76/5b258579f087be9b418f6bf63400ccdae1cd34bb4680bf26e701601ccb27/bbrf-1.0.7-py3-none-any.whl
Collecting slackclient==1.3.2 (from bbrf)
  Cache entry deserialization failed, entry ignored
Collecting docopt (from bbrf)
  Cache entry deserialization failed, entry ignored
[...]
Installing collected packages: chardet, certifi, urllib3, idna, requests, six, websocket-client, slackclient, docopt, bbrf
Successfully installed bbrf-1.0.7 certifi-2020.12.5 chardet-4.0.0 docopt-0.6.2 idna-2.10 requests-2.25.1 six-1.15.0 slackclient-1.3.2 urllib3-1.26.3 websocket-client-0.54.0

$ bbrf
Traceback (most recent call last):
  File "/home/xxx/.local/bin/bbrf", line 7, in <module>
    from bbrf import main
ModuleNotFoundError: No module named 'bbrf'