Code Monkey home page Code Monkey logo

nuclearpond's Introduction

Nuclear Pond

Nuclear Pond is used to leverage Nuclei in the cloud with unremarkable speed, flexibility, and perform internet wide scans for far less than a cup of coffee.

It leverages AWS Lambda as a backend to invoke Nuclei scans in parallel, choice of storing json findings in s3 to query with AWS Athena, and is easily one of the cheapest ways you can execute scans in the cloud.

Features

  • Output results to your terminal, json, or to S3
  • Specify threads and parallel invocations in any desired number of batches
  • Specify any Nuclei arguments just like you would locally
  • Specify a single host or from a file
  • Run the http server to take scans from the API
  • Run the http server to get status of the scans
  • Query findings through Athena for searching S3
  • Specify a custom nuclei and reporting configurations

Usage

Think of Nuclear Pond as just a way for you to run Nuclei in the cloud. You can use it just as you would on your local machine but run them in parallel and with however many hosts you want to specify. All you need to think of is the nuclei command line flags you wish to pass to it.

Setup & Installation

To install Nuclear Pond, you need to configure the backend terraform module. You can do this by running terraform apply or by leveraging terragrunt.

$ go install github.com/DevSecOpsDocs/nuclearpond@latest

Environment Variables

You can either pass in your backend with flags or through environment variables. You can use -f or --function-name to specify your Lambda function and -r or --region to the specified region. Below are environment variables you can use.

  • AWS_LAMBDA_FUNCTION_NAME is the name of your lambda function to execute the scans on
  • AWS_REGION is the region your resources are deployed
  • NUCLEARPOND_API_KEY is the API key for authenticating to the API
  • AWS_DYNAMODB_TABLE is the dynamodb table to store API scan states

Command line flags

Below are some of the flags you can specify when running nuclearpond. The primary flags you need are -t or -l for your target(s), -a for the nuclei args, and -o to specify your output. When specifying Nuclei args you must pass them in as base64 encoded strings by performing -a $(echo -ne "-t dns" | base64).

Commands

Below are the subcommands you can execute within nuclearpond.

  • run: Execute nuclei scans
  • service: Basic API to execute nuclei scans

Run

To run nuclearpond subcommand nuclearpond run -t devsecopsdocs.com -r us-east-1 -f jwalker-nuclei-runner-function -a $(echo -ne "-t dns" | base64) -o cmd -b 1 in which the target is devsecopsdocs.com, region is us-east-1, lambda function name is jwalker-nuclei-runner-function, nuclei arguments are -t dns, output is cmd, and executes one function through a batch of one host through -b 1.

$ nuclearpond run -h
Executes nuclei tasks in parallel by invoking lambda asynchronously

Usage:
  nuclearpond run [flags]

Flags:
  -a, --args string            nuclei arguments as base64 encoded string
  -b, --batch-size int         batch size for number of targets per execution (default 1)
  -f, --function-name string   AWS Lambda function name
  -h, --help                   help for run
  -o, --output string          output type to save nuclei results(s3, cmd, or json) (default "cmd")
  -r, --region string          AWS region to run nuclei
  -s, --silent                 silent command line output
  -t, --target string          individual target to specify
  -l, --targets string         list of targets in a file
  -c, --threads int            number of threads to run lambda functions, default is 1 which will be slow (default 1)

Custom Templates

The terraform module by default downloads the templates on execution as well as adds the templates as a layer. The variables to download templates use the terraform github provider to download the release zip. The folder name within the zip will be located within /opt. Since Nuclei downloads them on run we do not have to but to improve performance you can specify -t /opt/nuclei-templates-9.3.4/dns to execute templates from the downloaded zip. To specify your own templates you must reference a release. When doing so on your own repository you must specify these variables in the terraform module, github_token is not required if your repository is public.

  • github_repository
  • github_owner
  • release_tag
  • github_token

Retrieving Findings

If you have specified s3 as the output, your findings will be located in S3. The fastest way to get at them is to do so with Athena. Assuming you setup the terraform-module as your backend, all you need to do is query them directly through athena. You may have to configure query results if you have not done so already.

select
  *
from
  nuclei_db.findings_db
limit 10;

Advance Query

In order to get down into queries a little deeper, I thought I would give you a quick example. In the select statement we drill down into info column, "matched-at" column must be in double quotes due to - character, and you are searching only for high and critical findings generated by Nuclei.

SELECT
  info.name,
  host,
  type,
  info.severity,
  "matched-at",
  info.description,
  template,
  dt
FROM 
  "nuclei_db"."findings_db"
where 
  host like '%devsecopsdocs.com'
  and info.severity in ('high','critical')

Infrastructure

The backend infrastructure, all within terraform module. I would strongly recommend reading the readme associated to it as it will have some important notes.

  • Lambda function
  • S3 bucket
    • Stores nuclei binary
    • Stores configuration files
    • Stores findings
  • Glue Database and Table
    • Allows you to query the findings in S3
    • Partitioned by the hour
    • Partition projection
  • IAM Role for Lambda Function

nuclearpond's People

Contributors

jonathanwalker avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

nuclearpond's Issues

Getting a 403 when using the conventions in lambda.go but not when creating a session per AWS docs?

Hi... I was getting a 403 AccessDeniedException repeatedly when running nuclearpond in my environment.

I've successfully deployed via terraform and can test the lambda via CLI and Python boto3, getting expected results.

However, when running nuclearpond the invokeFunction was always failing and returning a 403 / AccessDenied.

I looked at https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/lambda-go-example-run-function.html and since I was getting desperate, I changed the code to more or less exactly match what was there.

I was a little surprised when that actually worked. I'm not sure the difference but using the following implementation for invokeFunction gave me no errors at all. Can anyone else reproduce this behavior or is this a "me" thing?

// Execute a lambda function and return the response
func invokeFunction(payload string, functionName string) (string, error) {
	// Create a new session

	sess := session.Must(session.NewSessionWithOptions(session.Options{
		SharedConfigState: session.SharedConfigEnable,
	}))

	// Create a Lambda service client.
	svc := lambda.New(sess, &aws.Config{
		Region: aws.String("us-east-1")},
	)

	// Create the input
	input := &lambda.InvokeInput{
		FunctionName: aws.String(functionName),
		Payload:      []byte(payload),
	}

[ERR] Could not update templates: Version string empty

Hey I was trying today to setup this cool project but i found a issue and i have no idea how to solve it... (Pretty new on terraform/aws envs...)

I really appreciate any help on this.

nuclearpond run -t google.com -r us-east-1 -f arn:aws:lambda:us-west-1:355250402967:function:nuclei-function  -a $(echo -ne "-t dns" | base64) -o cmd -b 1
  _   _                  _                           ____                        _
 | \ | |  _   _    ___  | |   ___    __ _   _ __    |  _ \    ___    _ __     __| |
 |  \| | | | | |  / __| | |  / _ \  / _` | | '__|   | |_) |  / _ \  | '_ \   / _` |
 | |\  | | |_| | | (__  | | |  __/ | (_| | | |      |  __/  | (_) | | | | | | (_| |
 |_| \_|  \__,_|  \___| |_|  \___|  \__,_| |_|      |_|      \___/  |_| |_|  \__,_|

                                                                  devsecopsdocs.com

2023/08/22 22:48:34 Running nuclei against the target devsecopsdocs.com
2023/08/22 22:48:34 Running with 1 threads
2023/08/22 22:48:35 Scan complete with output:

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.7

		projectdiscovery.io

[INF] nuclei-templates are not installed, installing...
[ERR] Could not update templates: Version string empty
[ERR] Could not read nuclei-ignore file: open /tmp/.config/nuclei/.nuclei-ignore: no such file or directory
[ERR] Could not find template 'dns': could not find template file: no such path found: dns
[INF] Using Nuclei Engine 2.8.7
[INF] Using Nuclei Templates 
[INF] Targets loaded for scan: 1
[INF] No results found. Better luck next time!
[FTL] Could not run nuclei: no valid templates were found

Cheers
Lola

When receiving a nuclei error, and -o json specified, nuclearpond does not correctly identify that nuclei has failed, and throws a JSON Parse error.

When running -t foobar ..... -o json, nuclearpond will throw an error, but the output is plain text e.g. "Nuclei could not find any templates".

This can't be parsed by the JSON parser of nuclearpond, and will throw an error at:

var prettyJSON bytes.Buffer
error := json.Indent(&prettyJSON, []byte(lambdaResponse.(string)), "", " ")
if error != nil {
log.Println("JSON parse error: ", error)
return
}

What was -json in 2.8.7 is now -jsonl

As title, arg has changed, so in 2.9.4, nuclearpond needs a patch to work properly. suggest just passing -j in all cases since it means the same in both versions.

When running "everything" it is extremely likely that you will run into a 15 min max lambda lifespan and timeout.

What is the best practices way to stripe jobs across so that I can run smaller chunks of "all" templates, while still making sure that I have a good "catch all" routine that won't be broken by later template updates?

The examples all show "-t dns", but for example if I were to use "-t http" it will fail because I need to also specify a subdirectory within http.

What is the best way to run "all of the templates" but avoid lambda timeouts?

Got an "panic: interface conversion: interface {} is nil, not string" error when scanning, which killed the whole job.

I saw the following when running a broad scan, I have not narrowed down the cause yet but figured I would get a dialogue going.
This was at approx host 900 out of a list of ~35000.

panic: interface conversion: interface {} is nil, not string

goroutine 230 [running]:
github.com/DevSecOpsDocs/nuclearpond/pkg/outputs.S3Output({0x0?, 0x0?})
	/home/vnc/nuclearpond/pkg/outputs/outputs.go:15 +0xfe
github.com/DevSecOpsDocs/nuclearpond/pkg/lambda.InvokeLambdas({{0xc000720c40, 0x1, 0x913c}, {0xc000700040, 0x2, 0x2}, {0x7fffce3b162b, 0x2}}, {0x7fffce3b15c1, 0x19}, ...)
	/home/vnc/nuclearpond/pkg/lambda/lambda.go:50 +0x1fd
github.com/DevSecOpsDocs/nuclearpond/pkg/core.ExecuteScans.func2()
	/home/vnc/nuclearpond/pkg/core/core.go:50 +0x5c
github.com/DevSecOpsDocs/nuclearpond/pkg/core.ExecuteScans.func1()
	/home/vnc/nuclearpond/pkg/core/core.go:32 +0x7a
created by github.com/DevSecOpsDocs/nuclearpond/pkg/core.ExecuteScans
	/home/vnc/nuclearpond/pkg/core/core.go:29 +0xad

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.