epi052 / recon-pipeline Goto Github PK

When tab completing scan commands, all possible commands are shown, not just installed commands.

Tab completion should only show available installed commands.

Describe the bug
The TKOSubsScan scan fails due to the file providers-data.csv being missing

INFO: [pid 114307] Worker Worker(salt=334832822, workers=1, host=GLaD0S, username=root, pid=114307, sudo_user=bad3r) running   TKOSubsScan(target_file=/tmp/tmpj58rd_um, db_location=/root/.local/recon-pipeline/databases/testing, results_dir=/home/bad3r/tmp/scan-results, exempt_list=, rate=500, interface=enp0s31f6, top_ports=100, ports=)
Error: open None/providers-data.csv: no such file or directory
Usage of /root/.local/recon-pipeline/tools/pipeline-go-workspace/bin/tko-subs:
  -data string
    	CSV file containing CMS providers' string for identification (default "providers-data.csv")
  -domain string
    	Domains separated by ,
  -domains string
    	List of domains to check (default "domains.txt")
  -githubtoken string
    	Github personal access token
  -herokuapikey string
    	Heroku API key
  -herokuappname string
    	Heroku app name
  -herokuusername string
    	Heroku username
  -output string
    	Output file to save the results (default "output.csv")
  -takeover
    	Flag to denote if a vulnerable domain needs to be taken over or not
  -threads int
    	Number of threads to run parallel (default 5)
ERROR: [pid 114307] Worker Worker(salt=334832822, workers=1, host=GLaD0S, username=root, pid=114307, sudo_user=bad3r) failed    TKOSubsScan(target_file=/tmp/tmpj58rd_um, db_location=/root/.local/recon-pipeline/databases/testing, results_dir=/home/bad3r/tmp/scan-results, exempt_list=, rate=500, interface=enp0s31f6, top_ports=100, ports=)
Traceback (most recent call last):
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/bad3r/git/recon-pipeline/pipeline/recon/web/subdomain_takeover.py", line 132, in run
    self.parse_results()
  File "/home/bad3r/git/recon-pipeline/pipeline/recon/web/subdomain_takeover.py", line 88, in parse_results
    with open(self.output_file, newline="") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/bad3r/tmp/scan-results/tkosubs-results/tkosubs.csv'

To Reproduce
Steps to reproduce the behavior:

1. follow installation instructions
2. create a database and attach it
3. run a TKOSubsScan

scan TKOSubsScan --results-dir /home/bad3r/tmp/scan-results --target secbytes.net --top-ports 100 --rate 500 --interface enp0s31f6 --verbose

Expected behavior
the file providers-data.csv should exist before the TKOSubsScan starts

Environment (please complete the following information):

• recon-pipeline version: 0.10.3
• python version: 3.8.3
• OS: Arch Linux

Additional context
I am not sure why the parent directory shows as None

open None/providers-data.csv: no such file or directory

initially i noticed the error while running a FullScan

detect OS; if it's kali, don't install exploitdb when running install all/exploitdb

First of all great tool and concept. I was following your blog and you have some great content.

When I try and run the ThreadedNmapScan I keep getting the following error. Regardless of system tried on Kali and Ubuntu 18.04. For some reason ThreadedNmapScan keeps failing. Even after following your blog and going in order.
Any thoughts?

RROR: [pid 13755] Worker Worker(salt=888760442, workers=1, host=jay-Z68MX-UD2H-B3, username=hack0r, pid=13755) failed    ThreadedNmapScan(target_file=/tmp/tmp6om43n4f, db_location=/home/hack0r/.local/recon-pipeline/databases/ffdff, results_dir=recon-results, exempt_list=, rate=1000, interface=enp4s0, top_ports=1000, ports=, threads=1)
Traceback (most recent call last):
  File "/home/hack0r/.local/share/virtualenvs/recon-pipeline-5UTm9lmr/lib/python3.7/site-packages/luigi/worker.py", line 184, in run
    raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: ParseMasscanOutput__home_hack0r__lo__enp4s0_f37f7cfb3e
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   ThreadedNmapScan__home_hack0r__lo__enp4s0_bb8aab3923   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 1 pending tasks possibly being run by other workers
DEBUG: There are 1 pending tasks unique to this worker
DEBUG: There are 1 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=888760442, workers=1, host=jay-Z68MX-UD2H-B3, username=hack0r, pid=13755) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 3 tasks of which:
* 1 complete ones were encountered:
    - 1 MasscanScan(...)
* 1 ran successfully:
    - 1 ParseMasscanOutput(...)
* 1 failed:
    - 1 ThreadedNmapScan(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

blocked by #83

• should return a LocalTarget pointing to its json results file
• should also return a SQLAlchemyTarget
• both returns should be in a dictionary

The repo is reporeted as HTML where it should be python.

The answer almost certainly lies here.

Is your feature request related to a problem? Please describe.
It's not.

Describe the solution you'd like
A query to api.recon.dev prior to running amass

Describe alternatives you've considered
None, it's an additional data source.

Describe the bug
After completing a full scan I wanted to view searchsploit results using the command:
view searchsploit-results
it returned a list error
EXCEPTION of type 'ValueError' occurred with message: 'list.remove(x): x not in list'

To Reproduce
Steps to reproduce the behavior:

1. install the tool basic on instructions
2. create a database and attach it
3. run a full scan on a domain
4. view searchsploit results after scan completions without errors

Expected behavior
view results of searchsploit

Traceback / Error Output
debugging traceback

[db-1] recon-pipeline> view searchsploit-results 
EXCEPTION of type 'ValueError' occurred with message: 'list.remove(x): x not in list'
To enable full traceback, run the following command: 'set debug true'
[db-1] recon-pipeline> set debug true
debug - was: False
now: True
[db-1] recon-pipeline> view searchsploit-results 
Traceback (most recent call last):
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/cmd2/cmd2.py", line 1657, in onecmd_plus_hooks
    stop = self.onecmd(statement, add_to_history=add_to_history)
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/cmd2/cmd2.py", line 2064, in onecmd
    stop = func(statement)
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/cmd2/decorators.py", line 223, in cmd_wrapper
    return func(cmd2_app, args)
  File "./pipeline/recon-pipeline.py", line 762, in do_view
    func(args)
  File "./pipeline/recon-pipeline.py", line 713, in print_searchsploit_results
    targets.remove(ipaddr.ipv4_address)
ValueError: list.remove(x): x not in list
EXCEPTION of type 'ValueError' occurred with message: 'list.remove(x): x not in list'

Environment (please complete the following information):

• recon-pipeline version: 0.10.3
• python version: 3.8.3
• OS: Arch Linux

Nuclei is a template based scanner. As such, not every template is appropriate to use on every single scan.

Ones that need to be excluded:

• brute-force
• payloads
• workflows

Workflows appear to simply wrap existing templates with add'l logic. We're probably double-tapping if we include them.

Command is probably something like

nuclei -t /tmp/stuff/nuclei-templates/cves -l TARGETS -silent -json

Need to explore -json vs. -json-requests

need to add searchsploit to the tool definitions

instructions for git install: https://www.exploit-db.com/searchsploit

Should be simple

go get github.com/tomnomnom/waybackurls

the binary gets dropped in ~/go/bin

webanalyze/tko-subs/subjack are all ok examples

Describe the bug
when installing tools I initially ran out of space. After removing some files to free up space I attempted to reinstall the tools by running the command install all which produced the error

[-] go queued
[-] gobuster queued
[-] subjack queued
[-] recursive-gobuster queued
[-] tko-subs queued
[-] luigi-service queued
[-] masscan queued
[-] amass queued
[-] webanalyze queued
[-] waybackurls queued
[-] seclists queued
[-] aquatone queued
[-] searchsploit queued
EXCEPTION of type 'EOFError' occurred with message: 'Ran out of input'
To enable full traceback, run the following command: 'set debug true'

To Reproduce
Steps to reproduce the behavior:

1. have limited space available
2. follow installation instructions

git clone https://github.com/epi052/recon-pipeline.git
cd recon-pipeline
pipenv install
pipenv shell

3. run ./recon-pipeline.py and install all tools install all

Expected behavior
Finish installing all tools without errors

Traceback / Error Output

Traceback (most recent call last):
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/cmd2/cmd2.py", line 1657, in onecmd_plus_hooks
    stop = self.onecmd(statement, add_to_history=add_to_history)
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/cmd2/cmd2.py", line 2064, in onecmd
    stop = func(statement)
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/cmd2/decorators.py", line 223, in cmd_wrapper
    return func(cmd2_app, args)
  File "./pipeline/recon-pipeline.py", line 343, in do_install
    self.do_install(tool)
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/cmd2/decorators.py", line 223, in cmd_wrapper
    return func(cmd2_app, args)
  File "./pipeline/recon-pipeline.py", line 347, in do_install
    tools = pickle.loads(persistent_tool_dict.read_bytes())
EOFError: Ran out of input

Environment (please complete the following information):

• recon-pipeline version: 0.10.3
• python version: 3.8.3
• OS Archlinux

add "Visualizing Tasks" section in docs that shows a pipeline in progress; i.e. expand upon the api reference to the status command and show what's available to be seen when using it.

blocked by #80

This ticket should not be the full implementation, simply the structure.

• should include docstring that conforms to project norms (see AmassScan for a good example)
• should inherit from ParseMasscanOutput
• should define runtime requirements
• should define requires, output, parse, and run methods with docstrings

check for default install location of seclists on kali; if it's there, create symlink to directory, mark tool as installed, and move on. don't install two copies of the same repo

Installing all dependencies on a (tested digitalocean) VPS does not take care of aquatone's chrome dependency.

Could look at installing headless chrome or phantomJS as part of the install command if it's not there.

masscan requires root privileges on ubuntu

allowing sudo to prompt without --verbose janks with the askpass prompt (i.e. in normal operations, the password doesn't make it through to sudo)

• it works with --verbose and normal askpass prompt
• preemptively running sudo -v and adding sudo to the masscan subprocess call works, as long as amass finishes within 15 minutes

proposed fix:

• add CAP_NET_RAW to the binary during installation. This will require root during install, but masscan will be able to execute without root from then on

blocked by #82 and #83

• should create subfolder for results
• should gather all web targets from db
• should utilize threading
• should run nuclei with each of the template categories we care about
• should call the parse method

While checking the newest PR from @GreaterGoodest, I tested installation on a fresh ubuntu 20.04 install. No issues found.

This one is up for thought/discussion. Nuclei templates have been seeing updates pretty frequently. In order to take advantage of them, we should issue

nuclei -update-templates -update-directory TOOLS_FOLDER

The question is when/how. It's relatively quick, we could spawn a thread and do it on startup silently. We could prompt the user on startup, but that feels invasive. Scan start w/ prompt might be ok...

Right now, i'm leaning toward a silent ninja update. Open for debate/thought tho

Is your feature request related to a problem? Please describe.
It is not related to a problem, strictly new functionality.

Describe the solution you'd like
I'd like to automate pulling information from the wayback machine/commoncrawl/virustotal using waybackurls. The information should then be parsed out into Endpoints and Targets (Targets as appropriate, i.e. if we find a new one, it should be added).

Once parsed, it would be cool to be able to filter out parameters so they can be sent to a wordlist for future use in fuzzing. This would probably be a second ticket/feature.

Describe alternatives you've considered
The alternative to waybackurls is gau. They both use commoncrawl and the wayback machine. gau uses alien vault's open threat exchange, and waybackurls uses virustotal. The deciding factor for me is that tomnomnom is a known quantity and produces a ton of good tooling, i.e. I decided on brand recognition.

Additional context
This would also allow us to pull all of the non-vendor javascript files found and parse them with LinkFinder (again, another separate ticket).

should:

• accept tool name as arg (similar to install)
• doesn't need the all keyword found w/in install

should remove:

• the tool itself
• the symlink in the .local/.../tools directory

Not immediately clear when dependencies are missing from a scan

"requires" function should check if dependency is installed.
threaded nmap should also "which nmap"

A nice quality of life improvement would be to add a reinstall command.

Should

• remove old installation
• call install for that tool

Blocked by:

• https://github.com/epi052/recon-pipeline/projects/4#card-36512722

Problem
I blow away my OS's a lot, it'd be nice to not have to set this tool up each time or worry about dependencies, even if it is very easy to set it up.

Solution
I'd like to be able to pull a Docker Image to my workstation whenever I need this tool.

Additional context
I'll start working on the Dockerfile and submit a PR for this. Just adding an issue mostly for tracking purposes.

I'm currently considering having a few volumes that would be passed through for this Image, such as the following:

• Tools: Directory where tools are installed so they're managed/can be added to outside of the container
• Scopes: Directory containing scope files so they're not blown away when the container is recreated
• Databases: Same reasoning as above, but for the databases

If you have anything else you want me to take into consideration while working on this, please mention it in this thread.

Install command

• GO111MODULE=on go get -u -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei

Dependencies

• [go]

blocked by #83

• should call meets_requirements
• should handle all necessary upstream arguments
• should handle nuclei specific arguments
• should return ParseMasscanOutput

need to update documentation to include the status command added in v0.8.0

Added documentation should be reflected in the README and docs.

Should outline the following:

• use of yaml tag functions
• the different options available in the config and when to use them (shell=true comes to mind)
• how to reuse the yaml "variables"

Should create a default/example/template tool definition and put it in the tools directory as part of the documentation effort.

When adding tools or changing functionality that affects the install/uninstall process, various versions of Ubuntu/Kali can have issues with these updates. These issues are hard to catch on a system that has already had the tools installed previously.

The proposed solution is to add CI stages that test the install process on clean versions of kali/ubuntu. Perhaps at various versions. This will work by executing a specific additional pytest, which looks for unexpected output.

The alternative is to continue manually testing all major changes on fresh VMs.

should contain a class named WaybackurlsScan that inherits from one of the two luigi Task classes used elsewhere.

see aquatone.py for an example of passing stdin to the subprocess, as that's what waybackurls expects.

currently, --target-file is the only way to specify targets to scan. This is cumbersome when there's only a single IP/domain to scan.

--target should take a single domain/ipv4/ipv6 address as an argument and scan that target without the user having to make a targetfile.

the easiest way would be to process the argument and then write the file for the user in the place where the pipeline expects it to be.

--target and --target-file should be mutually exclusive options

The download path for go assumes amd64 as the architecture. The result is that go is "installed" but can't run. Any tools that depend on go fail to install as well.

Currently, the installation process, default tool locations, package repositories, et al are all based on kali as the distro in use.

Would like to facilitate installation/usage on other distros, with ubuntu as the first target. This would mean changing

• default tool locations to ~/.recon or something similar (masscan/recursive-gobuster/aquatone/corscanner); go-based tools can stay in ~/go
• manually grabbing things installed by apt that aren't available on ubuntu (seclists comes to mind)
• updating user's path (probably just during recon-pipeline use)

install command should symlink installed tools final locations to recon-pipeline's tools directory. These symlinks should be the only thing used by scanners (implicit task).

Hi,

Scheduled 12 tasks of which:
* 1 complete ones were encountered:
    - 1 TargetList(target_file=/tmp/tmp64e09z6v, db_location=/root/.local/recon-pipeline/databases/1, results_dir=recon-results)
* 10 ran successfully:
    - 1 AmassScan(target_file=/tmp/tmp64e09z6v, db_location=/root/.local/recon-pipeline/databases/1, results_dir=recon-results, exempt_list=)
    - 1 AquatoneScan(...)
    - 1 GatherWebTargets(...)
    - 1 GobusterScan(...)
    - 1 MasscanScan(...)
    ...
* 1 failed:
    - 1 HTBScan(...)

This progress looks :( because there were failed tasks

I dont know why i get this error.

Error: argument --status-code: invalid choice: '403' (choose from )

Need to add black to the Pipfile as it's expected to be run for PRs.

Appears that amass is running from start to finish twice. Expected behavior is that it runs once to completion, then the second run sees existing results and immediately stops.

There's no mention of running sudo apt update prior to installing pipenv / running install command. If it's not run, there is a potential for failed installs of sub-commands that use apt. aquatone comes to mind when checking for headless-chrome.

This definitely applies to the README and almost certainly the docs.

Some files are dropped directly into results_dir and others are contained within a sub-directory. In the interest of organizing the results, the following changes should occur:

• all scanners should create a sub-directory and put their results within that sub-directory
• remove the target_file from the sub-directory naming convention (everything is in the results_dir already, no need to continue with the naming scheme)

Proposed structure follows

.
├── bitdiscovery
├── blacklist
└── recon-results
    ├── aquatone-results
    │   ├── amass.json
    ├── aquatone-results
    │   ├── aquatone_report.html
    │   ├── aquatone_session.json
    │   ├── aquatone_urls.txt
    │   ├── headers
    │   │   ├── http__104_20_60_51__42099b4af021e53f.txt
    │   ├── html
    │   │   ├── http__104_20_60_51__42099b4af021e53f.html
    │   └── screenshots
    │       ├── http__104_20_60_51__42099b4af021e53f.png
    ├── target-results
    │       ├── webtargets.txt
    │       ├── domains
    │       ├── ip6_addresses
    │       ├── ip4_addresses
    │       ├── subdomains
    ├── corscanner-results
    │       ├── corscanner.json
    ├── gobuster-results
    │   ├── gobuster.http_104.20.60.51:443.txt
    ├── masscan-results
    │       ├── masscan.json
    │       ├── masscan.parsed.pickle
    ├── nmap-results
    │   ├── nmap.104.20.60.51-tcp.gnmap
    │   ├── nmap.104.20.60.51-tcp.nmap
    │   ├── nmap.104.20.60.51-tcp.xml
    ├── searchsploit-results
    │   ├── searchsploit.104.20.60.51-tcp.txt
    ├── subjack-results
    │   ├── subjack.txt
    ├── tkosubs-results
    │   ├── tkosubs.csv
    ├── webanalyze-results
    │   ├── apps.json
    │   ├── webanalyze.http_104.20.60.51443.txt

Create issue template to formalize issue reporting / questions.

Should modify installation behavior so that the user's bashrc is not modified.

the purpose of this is to create symlinks in ~/.local/recon-pipeline/tools to each tool.

Currently the tool_paths dictionary points all over the file system. Additionally, it's awkward for the yaml file to query the tool_paths dict (which itself is a config). Each yaml definition should know where it should install itself.

All of the scans should only call the symlinks in .local/.../tools/... which are symlinks to the final location in the config

sub-commands:

• install
• uninstall
• reload (not sure this makes sense/is needed)
• reinstall

A tools command with sub-commands would align with other command/sub-command structures elsewhere in the project.

Documentation that will need updated:

• installation.rst
• visualization.rst
• scheduler.rst
• commands.rst

Blocked by:

• https://github.com/epi052/recon-pipeline/projects/4#card-36512507
• https://github.com/epi052/recon-pipeline/projects/4#card-36512722

In certain cases, specifying an absolute path for the target_file causes malformed paths to be produced by the pipeline.

Many of these are already fixed in the add-scan-tests branch, however they're not merged into master yet. Making note of the fact that the problem is known and actively being worked already.

The use of Luigi's central scheduler spins up a web server on port 8082 that shows the all queued tasks and their progress.

https://luigi.readthedocs.io/en/stable/central_scheduler.html

For those who would like to know where in the pipeline their tasks sit, the status command could open up the web browser to localhost:8082 for viewing.

Additionally, the same functionality can be exposed as a boolean flag to the scan command.

Notify user when unable to complete scan due to default results directory existing.

Additionally: prompt with option to overwrite this directory, or inform them of parameter to use to write somewhere else (--results-dir)

Should follow best practice and pin the version of library dependencies at what's currently working:

[dev-packages]
pytest==5.4.1
flake8==3.7.9
pre-commit==2.2.0
coverage==5.0.4
Sphinx==3.0.0
sphinx-argparse==0.2.5
sphinx-rtd-theme==0.4.3
sphinxcontrib-napoleon==0.7

[packages]
cmd2==1.0.1
luigi==2.8.12
SQLAlchemy==1.3.15
python-libnmap==0.7.0