[BUG] feroxbuster arbitrary paths despite specifying `--dont-extract-links` about feroxbuster HOT 7 CLOSED

howdy! if you're referencing the paths from your other issue #1007, those are the requests used to determine what the site uses as its 404 (whether it's a 404 or not).

the related code is here and the comments explain what's going on as well as some of the reasoning/decisions

from feroxbuster.

Hey. No this is not related to the other issue.
I just fired off another scan before commenting here. I'm seeing paths like:

/account
/oauth
/e
/s/
/5acab48dfbaf49b5bcf77d77a5edfae5

I assume the last one is also from feroxbuster intentionally along with the other like .admin and .htaccess. I'm using a wordlist of 3 words for test and none of these are present in my wordlist.

from feroxbuster.

correct, the last one is part of the heuristic check for determining 404s.

Could you take a look at the /robots.txt for your target? Those paths may be coming from there.

from feroxbuster.

You're right. /account and /oauth is in the robotx.txt of my target. I ran it across a large list so I can't confirm the other two

from feroxbuster.

morning! i just wrote a test to see whether or not --dont-extract-links controls requests to /robots.txt.

both in the code and in my test, /robots.txt is not requested when --dont-extract-links is used. Can you confirm that's the behavior you're seeing? The relevant code is below and requests for /robots.txt is gated behind the check for extract_links.

        if self.handles.config.extract_links && matches!(self.order, ScanOrder::Initial) {
            // check for robots.txt (cannot be in sub-directories, so limited to Initial)
            let mut extractor = ExtractorBuilder::default()
                .target(ExtractionTarget::RobotsTxt)
                .url(&self.target_url)
                .handles(self.handles.clone())
                .build()?;

            let result = extractor.extract().await?;
            extraction_tasks.push(extractor.request_links(result).await?)
        }

from feroxbuster.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from feroxbuster.

doing some housekeeping, feel free to reopen if this is still an issue.

from feroxbuster.