epi052 / feroxbuster Goto Github PK

very little is currently integration tested, any improvement here is beneficial. Will map out a lit of needs later

Describe the bug
Hello
Thank you for the awesome tool

I've been encountering this issue when fuzzing a url it abort the fuzzing and throw so unknow errors
To Reproduce
Steps to reproduce the behavior:

1. Downloaded the latest relases
2. Made an alias for automation looks like this

fastfuzz> # Defined in /root/.config/fish/functions/fastfuzz.fish @ line 1
          function fastfuzz
              feroxbuster --url $argv -w /opt/SecLists/Discovery/Web-Content/dic.txt -d 3 -x html,aspx,php,asp,log
          end

Works in fish
3. I run fastfuzz

Expected behavior
Bruteforce recursively and print out results

Traceback / Error Output

[>-------------------] - 8m     44744/1018686 85/s    http://95.163.33.203/
thread 'tokio-runtime-worker' panicked at 'Already joining!', /github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/indicatif-0.15.0/src/progress.rs:1035:13
stack backtrace:
   0:           0x679af8 - <unknown>
   1:           0x48d8ec - <unknown>
   2:           0x6791a6 - <unknown>
   3:           0x678baa - <unknown>
   4:           0x678431 - <unknown>
   5:           0x533e94 - <unknown>
   6:           0x53fbca - <unknown>
   7:           0x4e8dea - <unknown>
   8:           0x68d290 - <unknown>
   9:           0x68d025 - <unknown>
  10:           0x6873c5 - <unknown>
fish: Job 1, 'feroxbuster --url $argv -w /opt…' terminated by signal SIGABRT (Abort)

Environment (please complete the following information):

• feroxbuster version: 1.0.0
• OS [WSL 1]

Describe the solution you'd like
feroxbuster should look for its config in the same directory as the tool (current), ~/.config/feroxbuster, and cwd

Is your feature request related to a problem? Please describe.
I only want to see unfiltered responses in burp

Describe the solution you'd like
A --replay-proxy option that would only send responses identified as valid (basically not filtered out) to the proxy

Additional context
Thanks to @aringo and @hellor00t for the suggestion

need to add a -v to see it

thread 'tokio-runtime-worker' panicked at 'Already joining!', /home/epi/.cargo/registry/src/github.com-1ecc6299db9ec823/indicatif-0.15.0/src/progress.rs:1035:13

Describe the bug
Using the 2 provided commands to install on Mac gives a SHA256 mismatch

To Reproduce
Steps to reproduce the behavior:

1. brew tap tgotwig/feroxbuster
2. brew install feroxbuster

Expected behavior
A successful install

Traceback / Error Output
brew install feroxbuster
Updating Homebrew...
==> Installing feroxbuster from tgotwig/feroxbuster
==> Downloading https://raw.githubusercontent.com/epi052/feroxbuster/master/ferox-config.toml.example
######################################################################## 100.0%
Error: SHA256 mismatch
Expected: 70ace4e70c7f532cc4f7e7958106d035c62bd9d12a6a91de433b815f607911ba
Actual: d53171328e75472612470e337ec374376ede56631d12129aecb1cd29fefb69b8

Environment (please complete the following information):

• feroxbuster version: [ v1.4.0]
• OS [mac OS Catalina 10.15.7]

Describe the bug
When scanning a site that doesn't have a valid cert, the error message doesn't let the user know that the problem is certs. Instead, all that's shown is that ferox can't connect

To Reproduce
Steps to reproduce the behavior:

1. scan a site with an invalid cert
2. see ERROR heuristics::connectivity_test Could not connect to any target provided

Expected behavior
I expect a clear message notifying the user that the certificate is invalid

Environment (please complete the following information):

• feroxbuster version: v1.4.0

Additional context
Thanks to @Decap1tator for pointing out the issue

Is your feature request related to a problem? Please describe.
I'd like to distribute feroxbuster through a PPA. Long-term, i'd love to see it integrated into an official distro's repo (debian / ubuntu / kali).

Describe the solution you'd like

sudo add-apt-repository ppa:SOME_PPA
sudo apt update
sudo apt install feroxbuster

I'd like to have groups of options visually ... grouped together. Maybe not exactly these groups, but something along these lines. Currently clap doesn't offer this functionality but will in version 3.0.

Once clap 3.0 is released, this ticket can be completed.

Filter Options:
    -S, --filter-size <SIZE>...             Filter out messages of a particular size (ex: -S 5120 -S 4927,1970)
    -C, --filter-status <STATUS_CODE>...    Filter out status codes (deny list) (ex: -C 200 -C 401)
...

 Include Options:
    -s, --status-codes <STATUS_CODE>...     Status Codes to include (allow list) (default: 200 204 301 302 307 308 401)
...

Scan Options:
...

Client Options:
... 

Is your feature request related to a problem? Please describe.
I'd like to offer snap installation as an option for feroxbuster.

Describe the solution you'd like
sudo snap install feroxbuster

If anyone decides to work on this other than me, a snapcraft.yaml would satisfy this issue. From there I can handle publishing etc...

https://snapcraft.io/docs/rust-applications

update installation section in readme

Depending on a user's operating system's open file limit, the user may see a bunch of no file descriptors available errors. A user can increase the limit by editing limits.conf or using ulimit.

I'd like a description of the problem and both solutions to be included in the readme.

Definition of Done:

• entry in table of contents, something like Common Problems or similar
• sub-entry in table of contents to the above that indicates this particular problem (no file descriptors available)
• H2 heading that corresponds to the Common Problems or similar entry (include relevant emoji, warning symbol or something?)
• H3 heading that corresponds to the particular problem
• Under the H3 heading, a description of the problem and both limits.conf and ulimit solutions

Relevant discussions:

• #77 (comment)
• https://forum.hackthebox.eu/discussion/comment/85185

Hi

I just installed the tool via Homebrew on Mac

Now I need help to find the ferox-config.toml on my Mac, to add the wordlist path.

Thanks

update installation section in README

Ferox should support the ability to blacklist certain response codes, similar to the --hc flag in wfuzz. This would make it drastically easier to hide just one or two codes, versus the current alternative of having to whitelist all the codes you want to see.

Is your feature request related to a problem? Please describe.
When using --extract-links, it would be nice to have an option which only grabbed links from the original domain. I'm also not sure if it is starting to dir bust on other domains that are extracted? The output is unclear.

Describe the solution you'd like
A flag to limit the scope of the tool would be great. Also additional clarity in the ReadMe on if it starts busting new domains when using the --extract-links option would be great.

P.S. - Absolutely loving the tool! I think you've got a real edge on gobuster & ffuf with this one 👍. I've been sharing will all my colleagues! You've done some really great work on this!

When I run FeroxBuster with default settings, my output is full of status bars instead of all of the directories listed at the bottom. Is there a setting that I'm missing? I'm on Kali 2020, with the latest release 1.2.0

See Below:

Thank you

Is your feature request related to a problem? Please describe.

When you fuzz you need the most confortable output in order to analyze the results
When the request timeout it's shows in the output without any -v used

Describe the solution you'd like
I would like if possible a flag to supress this timeout urls and leave only the status code and useful information in the output

Screenshot

• missing config options
• better ToC
• comparison matrix
• etc...

write unit tests for create_urls

musl build on github's pipeline fails due to openssl dependency

When the tool created multiple recursive jobs, at one point, it has output a lot of errors and the CPU usage got to 100% at my machine and it was finally killed.

I have even tried to lower the threads count from 50 to 20 but as the recursive jobs increase I think it didn't really matter

The best way in my opinion to avoid this is to add an argument that takes a number to set the most jobs running at once, and add the new jobs to queue.

Other than that, your tool is awesome, thank you for the efforts!

With the inclusion of console, feroxbuster has two crates that can color the terminal. I use console for other things, so should swap the coloring to use console as well, then remove ansi_term.

Is your feature request related to a problem? Please describe.
Due to how MultiProgress handles printing when not in a user_attended shell, there is no simple way to capture the output generated from -v+.

Describe the solution you'd like
A way for those logs to be captured in a file. I think a --logfile makes sense, but am open to suggestions.

A blanket issue for test coverage improvements

bars are printed over each other, but shouldn't be

A common problem I run into is that some sites have issues when supporting too many concurrent connections, or too high a rate of connections- of course every site has its breaking point- but read my next paragraph for more detail on this. This can be mitigated by tuning feroxbuster with -t and -L for each individual target, of course.

The problem comes in when doing testing across a large amount of sites at once, using, e.g. GNU parallel. If you are performing testing against a medium or large organization with many websites, sometimes you'll need to batch a large set of commands due to testing time constraints, and it won't be practical to test and tune the -t and -L setting for each individual site, since they can vary quite a bit within a large set. Consider for this example a list of 1000 or more sites.

A nice feature would be to either:

1. (Simple Solution) Simply bail out after n connection failures
2. (Complex / Better Solution) Tune the threads and/or concurrent connections setting dynamically, based on the occurrence of connection failures

Some workarounds here:

1. As I mentioned, manually testing each site and having a per-site -t and -L setting; this is prohibitively expensive in terms of time during a large-scale test
2. Using a very conservative value across all sites; this is detrimental to the large amount of sites that can handle (in many cases) many multiples of that conservative setting, slowing the entire testing run of the entire batch down significantly

This may be beyond the scope of what you would like to implement and maintain within feroxbuster, but for me, it would be a very useful feature.

Curious what you think about this

Thanks, I appreciate your development on this tool. I haven't seen a public tool that performs as well as feroxbuster, with such flexibility and robust and advanced features since skipfish- which is no longer maintained and never really had a happy medium between "way too agressive" and "completely limited in its findings"

writing to output file is not async, make it so.

in doing so, i'd prefer the output file to be sorted by url before the program exits via a cleanup section (indicating to the user that scans are done and we're performing clean up things)

I would like to create Brew formulas for MacOS & Linux to make installation easier 🙂

• put url at the end of the line instead of the middle
• give size a normalized size with string formatting
• decide whether to keep canonical status code reason in output or shrink to just the 2xx/3xx/4xx etc codes

banner and bars should print to stderr
url reporting should print to stdout

the indicatif library is making this difficult

Is your feature request related to a problem? Please describe.
Nope. Refer to previous post on the love of this script.

Describe the solution you'd like
To change (or have a different -q flag) for:

Describe alternatives you've considered
Using -q works but I would prefer to have all the information but the time visible. Or to show progress only once it has found a URL or String.

Additional context
My reasoning for this is I use TMUX and I have it set to notify me on panes when something appears in the HUD. Useful for notifying me well-running an HTTP server and picking up the request... or doing further enumeration well exploring other ports. Having this feature would mean I would only get a notification when a URL/string was found.

Is your feature request related to a problem? Please describe.
Per this issue, excessive CPU usage is a possibility given the current implementation of recursion.

Describe the solution you'd like
Limit the number of recursive scans by using a consumer/producer w/ queue system. The limit should have a sane default and be configurable from the command line/config file.

https://github.com/epi052/feroxbuster/blob/master/.github/workflows/build.yml needs to be updated such that the x86_64 build creates a tar.gz file as well. This will be used for homebrew installs.

An example already exists in the build-macos job. https://github.com/epi052/feroxbuster/blob/master/.github/workflows/build.yml#L62

This issue is the mirror of that for 64 bit linux builds.

stream the wordlist to scanning as i read it in, instead of blocking

Is your feature request related to a problem? Please describe.
As discussed here and here, low limits on number of open files allowed by the OS can result in spurious errors reported to the user.

Describe the solution you'd like
Use setrlimit syscall to adjust the number of open files allowed, if necessary.

• The limit should be reset to normal once complete
• will likely need to inspect the hard limit with getrlimit in order to know the max adjustment allowed for an unprivileged user
• consider notifying the user of the change

Is your feature request related to a problem? Please describe.
When using --extract-links, it would be nice to have an option which only grabbed links from the original domain. I'm also not sure if it is starting to dir bust on other domains that are extracted? The output is unclear.

Describe the solution you'd like
A flag to limit the scope of the tool would be great. Also additional clarity in the ReadMe on if it starts busting new domains when using the --extract-links option would be great.

P.S. - Absolutely loving the tool! I think you've got a real edge on gobuster & ffuf with this one 👍. I've been sharing will all my colleagues! You've done some really great work on this!

-q should create hidden progress bars

For both linux binaries, strip -s should be run on the binary before upload in order to reduce final binary size.

If there is a similar command for macos, include that in the macos build.

Describe the bug
In certain situations, duplicate scans are kicking off against the same directory.

Expected behavior
A single scan per directory.

Environment (please complete the following information):

• feroxbuster version: v1.0.1

Additional context
A HashSet of scanned urls would likely solve the problem.

start scan against URL-1
add URL-1 to the url-set
...
new URL found to scan
if new URL in url-set -> do nothing
else -> scan

The ability to white/blacklist based off of characters/line count on the response page would be greatly appreciated. Sometimes you won't find what you want from a response code, but will from a char or line count. Both features are in wfuzz for comparison.

Originally reported by @LMAY75

Describe the solution you'd like
As valid 2xx responses are found, examine their contents for additional files/directories.

All directories found should be added to new recursive scans, as long as they don't exceed the recursion depth limit. If a newly discovered directory is found and exceeds the limit, the user should still be notified.

Extracting links should be an opt-in feature, as there's a cost for the additional coverage provided.

Additional context
The feature branch for this is tracked in the linked pull request

Describe the solution you'd like
I'd like the ability to pause feroxbuster mid-scan via keyboard input and then be able to resume the scan, also via keyboard input

originally suggested by @Flangyver

Describe the bug
When -x is used, the scanner only increments the counter when a 'base' request is made. It exits shortly after the number of requests reach roughly the length of the wordlist. This indicates that requests generated from extensions aren't incrementing the progress bar as expected.

To Reproduce
Steps to reproduce the behavior:

1. feroxbuster -u SOMEURL -x php

Take note of the # of words in the wordlist and the # of requests sent. Expect exit shortly after requests reach # of words.

Expected behavior
All requests should increment progress the bar, not only the base requests.

Environment (please complete the following information):

• feroxbuster version: 1.0.0 - 1.1.1
• OS:
  • kali 2020.2
  • ubuntu 20.04

Is your feature request related to a problem? Please describe.
Nope; actually I came across this last week and love it.

Describe the solution you'd like
The ability to cut a thread on a scan. So for instance, if I find a hidden CMS solution on a port at /wordpress/ and the scan begins scanning the /wordpress/ directory to stop the scan of the original directory. I get I could cancel or start a new instance but with application searching, you could filter your recursive searching to cut down on time. I guess you could also pipe it into an additional scan?

Describe alternatives you've considered
Cancelling the current scan and starting a new one.

Additional context
None.

Is your feature request related to a problem? Please describe.
I'd prefer that folks are notified of new releases when they're available.

Describe the solution you'd like
A user runs feroxbuster, if the user's version is behind the current release, notify the user (probably in the banner)

Describe alternatives you've considered
The alternative to always checking would be a --update flag or similar, however, I prefer the auto check

Hi, recently, I am testing to see what can feroxbuster do, from my local machine. But, I think that "🎯", "🚀", ... will make feroxbuster like a funny joke script rather than an advanced fuzzer, a complex fuzzing project. Should we delete it?

this is more of a 'can we' issue

can the lazy_static call be housed inside an initialize function for conformity?

logging dorks the progress bar output. the link below can be used as a solution template

https://github.com/getsentry/sentry-cli/blob/master/src/utils/logging.rs