fyoorer / shadowclone Goto Github PK

Unleash the power of cloud

License: Apache License 2.0

Dockerfile 15.85% Python 68.78% Roff 15.37%

distributed-computing cloud lambda-functions recon bugbounty aws serverless

shadowclone's Introduction

ShadowClone

ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete.

You can make full use of the Free Tiers provided by cloud providers and supercharge your mundane cli tools with shadow clone jutsu (Naruto style)!

Installation

Please visit the wiki for installation and intial configuration instructions

Usage

⚡ python shadowclone.py -h
usage: shadowclone.py [-h] -i INPUT [-s SPLITNUM] [-o OUTPUT] -c COMMAND

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
  -s SPLITNUM, --split SPLITNUM
                        Number of lines per chunk of file
  -o OUTPUT, --output OUTPUT
  -c COMMAND, --command COMMAND
                        command to execute
  --no-split NOSPLIT    File to be used without splitting

How it works

We create a container image during the initial setup and register it as a runtime for our function in AWS/GCP/Azure whatever. When you execute ShadowClone on your computer, instances of that container are activated automatically and are only active for the duration of its execution. How many instances to activate is dynamically decided at runtime depending on the size of the input file provided and the split factor. The input is then split into chunks and equally distributed between all the instances to execute in parallel. For example, if your input file has 10,000 lines and you set the split factor to 100 lines, then it will be split into 100 chunks of 100 lines each and 100 instances will be run in parallel!

Features

Extremely fast
No need to maintain a VPS (or a fleet of it :))
Costs almost nothing per month
- Compatible with free tiers of most cloud services
Cloud agnostic
- Same script works with AWS, GCP, Azure etc.
Supports upto 1000 parallel invocations
Dynamically decide the number of invocations
Run any tool in parallel on the cloud
Pipe output to other tools

Comparison

This tool was inspired by the awesome Axiom and Fleex projects and goes beyond the concept of VPS for running the tools by using serverless functions and containers.

Features	Axiom/Fleex	ShadowClone
Instances	10-100s*	1000s
Cost	Per instance/per minute	Mostly Free**
Startup Time	4-5 minutes	2-3 seconds
Max Execution Time	Unlimited	15 minutes
Idle Cost	$++	Free
On Demand Scalability	No	∞

*Most cloud providers do not allow spinning up too many instances by default, so you are limited to around 10-15 instances at max. You have to make a request to the support to increase this number.

** AWS & Azure allow 1 million invocations per month for free. Google allows 2 million invocations per month for free. You will be charged only if you go above these limits

Demo

DNS Bruteforcing using a 43mb file - 34 seconds

Running httpx on 94K subdomains in 1 min

References

Lithops documentation

Free Tiers

Cloud Provider	Free Allowance	Link
Google Functions	2 Million invocations, 400,000 GB-seconds per month	Google Cloud Free Program
AWS Lambda	1 Million invocations, Up to 3.2 million seconds of compute time per month	Free Cloud Computing Services - AWS Free Tier
Azure Functions	1 Million invocations	Microsoft Azure Free Services

Obviously, you can make any number of function invocations per month. The table above only shows how many invocations are free.

Similar Tools

Support

If you like ShadowClone and would like to show some appreciation, send me the bugs you find using this tool ;)

Or you can buy me a coffee :)

Disclaimer

This tool is designed as a proof-of-concept implementation and it's usage is intended for educational purpose only. Usage for attacking targets without prior mutual consent is illegal. It's the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program.

shadowclone's People

Contributors

Stargazers

Watchers

Forkers

taku65maku attacker-codeninja mahmoudimus emptysec1 ehsandeep vysecurity bbhunter zeeshan0346 minkione h121h sasqwatch silentsoul04 optionalg shantanu561993 ausecwa danis24 0xsojalsec zxpower ringwraith dew2120 daja19 elchappo cyberlixio marmodsec tr3ss mantissts govi123govi slooppe atlassion medmahmoudi26 booboohq evcuq4hggjd74lhz josephgregg 0xwise64 m3chboy e1abrador vinnyvinoth warrdaddy mswell matanber 0xdexter0us srand2 blacknesses minight reewardius maycon ajaymagar resenhacyber wall6e viniciushammett selomoscar2 houtaro001 peff1235 puzzlepeaches asadanwar90 mathdeziel nullze danzee1 hackbynight ongyuann tazwinator daemonsgiants ndbrain affilares kelly-blue ramosslyz gpxlnx webclinic017 balance0033 linuxkd dalaho-bangin santaonbeach findthehead gujjuboy10x00 cyberkid02 ninjayoto singhajeet79 sanu1999 sucre92 osamahamad saibashoejaa thisedgarmarquez gautam-hacxie tester-byte nishantbhagat57 beerandgin road5 dztest1 peachycloudsecurity kholi0o hawkpubg dz-test b1luuu

shadowclone's Issues

Could not execute the runtime. matrix transpose with ndim < 2 is undefined Error

I keep recieving the following error when executing anything on shadowclone.py matrix transpose with ndim < 2 is undefined - Any help would be appreciated

(env) root@ubuntu-16gb-hel1-2:~/ShadowClone# python shadowclone.py -i test.txt --split 1 -o testbro.txt -c "/go/bin/httpx -l {INPUT}"
2024-06-17 01:49:21,617 [INFO] Input file is small enough. Skipping upload.
2024-06-17 01:49:21,617 [INFO] Splitting input file into chunks of 1 lines
2024-06-17 01:49:21,617 [INFO] Scan ID: YSL1JmQu
2024-06-17 01:49:21,620 [INFO] config.py:134 -- Lithops v2.8.0
2024-06-17 01:49:21,625 [INFO] aws_s3.py:60 -- S3 client created - Region: us-east-1
2024-06-17 01:49:22,086 [INFO] aws_lambda.py:94 -- AWS Lambda client created - Region: us-east-1
2024-06-17 01:49:22,087 [INFO] invokers.py:108 -- ExecutorID 2382b4-0 | JobID M000 - Selected Runtime: sc-runtime - 512MB
2024-06-17 01:49:22,089 [ERROR] Could not execute the runtime.
matrix transpose with ndim < 2 is undefined

Error while running the task

Python 3.9.16
The test input file contained only 100000 lines.

Error when running build command

Hi!

I've been trying to setup the tool on my Debian 10 box and getting the error when running the command:

lithops runtime build sc-runtime -f ~/tools/ShadowClone/Dockerfile

Error snapshot:

Python version : Python 3.10.6

Can it be run on our local machine with local network?

I know these tools are only used for cloud-based. But, Can it be run on our local machine with the local network? Or any system like this?

Delete me

delete

Docker build issue!

Please help!!

httpx running error

I am using this command python shadowclone.py -i domains.txt -c "/go/bin/httpx -l {INPUT}" and getting this error

Encountered a bad command exit code!

Command: '/go/bin/httpx -l /tmp/infile'

Exit code: 127

Stdout: already printed

Stderr: already printed

nuclei and subfinder working fine only getting this error in httpx!

UPDATE: Fixed all errors Message me on Discord I'll send you starting to ending commands to properly install this tool, because official wiki is not updated. USERNAME: ba1van7

Installation Error: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

Im Getting this error file installing shadow clone...

(env) blackbox@groot:~/tools/ShadowClone$ lithops runtime build sc-runtime -f Dockerfile
2023-05-21 16:58:48,375 [INFO] config.py:134 -- Lithops v2.8.0
2023-05-21 16:58:48,375 [DEBUG] config.py:93 -- Loading configuration from /home/blackbox/.lithops/config
2023-05-21 16:58:48,406 [DEBUG] config.py:186 -- Loading Serverless backend module: aws_lambda
2023-05-21 16:58:48,496 [DEBUG] aws_lambda.py:52 -- Creating AWS Lambda client
2023-05-21 16:58:48,496 [DEBUG] aws_lambda.py:65 -- Creating Boto3 AWS Session and Lambda Client
2023-05-21 16:58:49,699 [INFO] aws_lambda.py:94 -- AWS Lambda client created - Region: us-east-1
2023-05-21 16:58:49,701 [INFO] aws_lambda.py:342 -- Building runtime sc-runtime from Dockerfile
2023-05-21 16:58:49,702 [DEBUG] utils.py:230 -- Creating function handler zip in lithops_lambda.zip
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=sc-runtime&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied
Traceback (most recent call last):
  File "/home/blackbox/tools/ShadowClone/env/bin/lithops", line 8, in <module>
    sys.exit(lithops_cli())
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/lithops/scripts/cli.py", line 456, in build
    compute_handler.build_runtime(runtime_name, file, ctx.args)
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/lithops/serverless/serverless.py", line 66, in build_runtime
    self.backend.build_runtime(runtime_name, file, extra_args)
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 354, in build_runtime
    utils.run_command(cmd)
  File "/home/blackbox/tools/ShadowClone/env/lib/python3.10/site-packages/lithops/utils.py", line 713, in run_command
    sp.check_call(cmd.split())
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/docker', 'build', '-t', 'sc-runtime', '-f', 'Dockerfile', '.']' returned non-zero exit status 1.

And when i try to run this as root

(env) blackbox@groot:~/tools/ShadowClone$ sudo lithops runtime build sc-runtime -f Dockerfile
sudo: lithops: command not found

Exception within runnerjob.py that prevents execution of jobs

Hello, there was an issue while running shadowclone.py to resolve dns records. From the terminal output below, you can see that 4 functions activated however it looks like the service was unable to execute the run time. Upon inspection of poll output, there was an exception within runnerjob.py.

Exception

    Traceback (most recent call last):
      File "/function/lithops/worker/jobrunner.py", line 228, in run
        result = func(**data)
      File "shadowclone.py", line 94, in execute_command
    AttributeError: 'NoneType' object has no attribute 'replace'

Terminal Output

user@Administrators-MacBook-Pro-7 ~/arsenal/ShadowClone                                                                                                                                                              [17:10:40]
(recon-venv) > $ sudo python shadowclone.py -i  ~/Desktop/compiled.txt  --split 3000000 -o ~/Desktop/test.txt -c "/go/bin/puredns resolve --resolvers ~/Desktop/resolvers.txt"                                         [±main ●●]
2023-02-11 17:11:33,138  [INFO] Splitting input file into chunks of 3000000 lines
2023-02-11 17:11:35,729  [INFO] Uploading chunks to storage
2023-02-11 17:14:08,275 [INFO] lithops.config -- Lithops v2.6.0
2023-02-11 17:14:08,301 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: us-west-1
2023-02-11 17:14:08,804 [INFO] lithops.serverless.backends.aws_lambda.aws_lambda -- AWS Lambda client created - Region: us-west-1
2023-02-11 17:14:08,805 [INFO] lithops.invokers -- ExecutorID f049ae-0 | JobID M000 - Selected Runtime: lithops_v2-6-0_gupt/sc-runtime - 512MB
2023-02-11 17:14:10,253 [INFO] lithops.invokers -- ExecutorID f049ae-0 | JobID M000 - Starting function invocation: execute_command() - Total: 4 activations
2023-02-11 17:14:10,292 [INFO] lithops.invokers -- ExecutorID f049ae-0 | JobID M000 - View execution logs at /private/tmp/lithops/logs/f049ae-0-M000.log
2023-02-11 17:14:10,295 [INFO] lithops.wait -- ExecutorID f049ae-0 - Getting results from 4 function activations

    0%|                                                                                                                                                                                                                     | 0/4

2023-02-11 17:14:15,356 [INFO] lithops.executors -- ExecutorID f049ae-0 - Cleaning temporary data
2023-02-11 17:14:15,366  [ERROR] Could not execute the runtime.

Poll output

user@Administrators-MacBook-Pro-7 ~/arsenal/ShadowClone                                                                                                                                                              [17:15:27]
(recon-venv) > $ lithops logs poll                                                                                                                                                                                      [±main ●●]
Activation: 'python3' (6157671bc704)
[
    2023-02-11 15:59:51,378 [INFO] lithops.worker.handler -- Lithops v2.6.0 - Starting Localhost execution
    2023-02-11 15:59:51,379 [INFO] lithops.worker.handler -- Execution ID: c0571d-0-A000/00000
    2023-02-11 15:59:51,379 [INFO] lithops.storage.backends.localhost.localhost -- Localhost storage client created
    2023-02-11 15:59:51,385 [INFO] lithops.worker.jobrunner -- Going to execute 'hello()'
    ---------------------- FUNCTION LOG ----------------------
    ----------------------------------------------------------
    2023-02-11 15:59:51,386 [INFO] lithops.worker.jobrunner -- Success function execution
    2023-02-11 15:59:51,387 [INFO] lithops.worker.jobrunner -- Storing function result - Size: 41.0B
    2023-02-11 15:59:51,388 [INFO] lithops.worker.jobrunner -- Process finished
]

Activation: 'python38' (e7443555-f092-40ea-81a5-b4e863a77f7f)
[
    2023-02-12 00:47:42,450 [INFO] lithops.worker.handler -- Lithops v2.6.0 - Starting AWS Lambda execution
    2023-02-12 00:47:42,450 [INFO] lithops.worker.handler -- Execution ID: ec9eb2-0-A000/00000
    2023-02-12 00:47:42,472 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: us-west-1
    2023-02-12 00:47:42,554 [INFO] lithops.worker.jobrunner -- Going to execute 'hello()'
    ---------------------- FUNCTION LOG ----------------------
    ----------------------------------------------------------
    2023-02-12 00:47:42,554 [INFO] lithops.worker.jobrunner -- Success function execution
    2023-02-12 00:47:42,555 [INFO] lithops.worker.jobrunner -- Storing function result - Size: 41.0B
    2023-02-12 00:47:42,595 [INFO] lithops.worker.jobrunner -- Process finished
]

Activation: 'lithops_v2-6-0_gupt/sc-runtime' (9c11515c-055e-49d6-87b3-94138a4c106e)
[
    2023-02-12 01:01:44,053 [INFO] lithops.worker.handler -- Lithops v2.6.0 - Starting AWS Lambda execution
    2023-02-12 01:01:44,054 [INFO] lithops.worker.handler -- Execution ID: c94cbf-0-M000/00000
    2023-02-12 01:01:44,061 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: us-west-1
    2023-02-12 01:01:45,015 [INFO] lithops.worker.jobrunner -- Getting dataset from aws_s3://runtime-shadowclone/959adbc3-b543-4015-ae2b-90762961b742
    2023-02-12 01:01:45,121 [INFO] lithops.worker.jobrunner -- Going to execute 'execute_command()'
    ---------------------- FUNCTION LOG ----------------------
    ----------------------- EXCEPTION !-----------------------
    Traceback (most recent call last):
      File "/function/lithops/worker/jobrunner.py", line 228, in run
        result = func(**data)
      File "shadowclone.py", line 94, in execute_command
    AttributeError: 'NoneType' object has no attribute 'replace'
    ----------------------------------------------------------
    2023-02-12 01:01:47,095 [INFO] lithops.worker.jobrunner -- Process finished
]

Activation: 'lithops_v2-6-0_gupt/sc-runtime' (fbcb2251-f728-479f-b263-6050f2641d2f)
[
    2023-02-12 01:01:44,392 [INFO] lithops.worker.handler -- Lithops v2.6.0 - Starting AWS Lambda execution
    2023-02-12 01:01:44,392 [INFO] lithops.worker.handler -- Execution ID: c94cbf-0-M000/00003
    2023-02-12 01:01:44,412 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: us-west-1
    2023-02-12 01:01:45,370 [INFO] lithops.worker.jobrunner -- Getting dataset from aws_s3://runtime-shadowclone/337ac1fb-35b3-4fd3-8af3-4d3beb75a27d
    2023-02-12 01:01:45,493 [INFO] lithops.worker.jobrunner -- Going to execute 'execute_command()'
    ---------------------- FUNCTION LOG ----------------------
    ----------------------- EXCEPTION !-----------------------
    Traceback (most recent call last):
      File "/function/lithops/worker/jobrunner.py", line 228, in run
        result = func(**data)
      File "shadowclone.py", line 94, in execute_command
    AttributeError: 'NoneType' object has no attribute 'replace'
    ----------------------------------------------------------
    2023-02-12 01:01:47,507 [INFO] lithops.worker.jobrunner -- Process finished
]

deploy error

(env) root@zyz:~/ShadowClone# lithops runtime create recon --memory 512 --timeout 800
2022-12-06 00:33:19,115 [INFO] lithops.config -- Lithops v2.5.8
2022-12-06 00:33:19,115 [DEBUG] lithops.config -- Loading configuration from /root/.lithops/config
2022-12-06 00:33:19,133 [DEBUG] lithops.config -- Loading Serverless backend module: aws_lambda
2022-12-06 00:33:19,192 [DEBUG] lithops.config -- Loading Storage backend module: aws_s3
2022-12-06 00:33:19,193 [INFO] lithops.scripts.cli -- Creating new lithops runtime: recon
2022-12-06 00:33:19,193 [DEBUG] lithops.storage.backends.aws_s3.aws_s3 -- Creating S3 client
2022-12-06 00:33:19,290 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: ap-southeast-1
2022-12-06 00:33:19,290 [DEBUG] lithops.serverless.backends.aws_lambda.aws_lambda -- Creating AWS Lambda client
2022-12-06 00:33:19,290 [DEBUG] lithops.serverless.backends.aws_lambda.aws_lambda -- Creating Boto3 AWS Session and Lambda Client
2022-12-06 00:33:20,206 [INFO] lithops.serverless.backends.aws_lambda.aws_lambda -- AWS Lambda client created - Region: ap-southeast-1
2022-12-06 00:33:20,207 [DEBUG] lithops.serverless.backends.aws_lambda.aws_lambda -- Creating new Lithops lambda function: lithops_v2-5-8_gzmx_recon_512MB
Traceback (most recent call last):
  File "/root/ShadowClone/env/lib/python3.8/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 371, in create_runtime
    response = self.ecr_client.describe_images(repositoryName=repo_name)
  File "/root/ShadowClone/env/lib/python3.8/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/root/ShadowClone/env/lib/python3.8/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.RepositoryNotFoundException: An error occurred (RepositoryNotFoundException) when calling the DescribeImages operation: The repository with name 'lithops_v2-5-8_gzmx/recon' does not exist in the registry with id '866113468777'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/ShadowClone/env/bin/lithops", line 8, in <module>
    sys.exit(lithops_cli())
  File "/root/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/root/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/root/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/root/ShadowClone/env/lib/python3.8/site-packages/lithops/scripts/cli.py", line 434, in create
    runtime_meta = compute_handler.create_runtime(name, mem, timeout=to)
  File "/root/ShadowClone/env/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 73, in create_runtime
    return self.backend.create_runtime(runtime_name, memory, timeout=timeout)
  File "/root/ShadowClone/env/lib/python3.8/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 376, in create_runtime
    raise Exception('Runtime {} is not deployed to ECR'.format(runtime_name))
Exception: Runtime recon is not deployed to ECR

I thought the problem was with the "role" but I have adjusted what you gave on the wiki and it still has an error

Please add Instructions for setting up GCP

Hello,
Please add the Instructions for setting up GCP.

Function stuck in pending state

It worked at the first go.

Then I've added some other tooling to the Dockerfile and used this command to rebuild the Docker:

lithops runtime build sc-runtime -f Dockerfile

Since then, I'm not able anymore to run the program. Even Lithops test fails me:

lithops test
2022-01-24 10:46:09,925 [INFO] lithops.config -- Lithops v2.5.8
2022-01-24 10:46:10,081 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: eu-west-3
2022-01-24 10:46:10,124 [INFO] lithops.serverless.backends.aws_lambda.aws_lambda -- AWS Lambda client created - Region: eu-west-3
2022-01-24 10:46:10,125 [INFO] lithops.invokers -- ExecutorID d5971d-0 | JobID A000 - Selected Runtime: python38 - 512MB
2022-01-24 10:46:10,218 [INFO] lithops.invokers -- Runtime python38 with 512MB is not yet installed
2022-01-24 10:46:10,311 [INFO] lithops.serverless.backends.aws_lambda.aws_lambda -- Creating default lambda layer for runtime python38
Traceback (most recent call last):
  File "/root/tools/ShadowClone/test/bin/lithops", line 8, in <module>
    sys.exit(lithops_cli())
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/lithops/scripts/cli.py", line 168, in test_function
    fexec.call_async(hello, username)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/lithops/executors.py", line 205, in call_async
    runtime_meta = self.invoker.select_runtime(job_id, runtime_memory)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/lithops/invokers.py", line 121, in select_runtime
    runtime_meta = self.compute_handler.create_runtime(self.runtime_name, runtime_memory, runtime_timeout)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 73, in create_runtime
    return self.backend.create_runtime(runtime_name, memory, timeout=timeout)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 413, in create_runtime
    layer_arn = self._create_layer(runtime_name)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 200, in _create_layer
    response = self.lambda_client.invoke(
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/root/tools/ShadowClone/test/lib/python3.8/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceConflictException: An error occurred (ResourceConflictException) when calling the Invoke operation: The operation cannot be performed at this time. The function is currently in the following state: Pending

Already tried:

Remove runtime reference in config.py
Remove lambda function manually in console
Remove S3 bucket and create a new one and add new one in config
Changing AWS region

Would really appreciate some guidance in how to proceed when editing the Dockerfile

Amazon Lamba Execution time 5 minutes

Error:

Function exceeded maximum time of 295 seconds and was killed

However, the main README.md mentions the maximum time is 15 minutes which should be 3 times this amount at around 900 seconds.

pickle_db

Hi, what can I put for Pickle_db or where can I find the bucket-hash.db?

python shadowclone.py -i /home/kali/range.txt --split 100 -o test.txt -c "/go/bin/httpx -l {INPUT}"
2022-10-11 23:07:27,214 [INFO] Splitting input file into chunks of 100 lines
2022-10-11 23:07:27,219 [INFO] Uploading chunks to storage
Traceback (most recent call last):
File "/home/kali/Tools/ShadowClone/shadowclone.py", line 128, in
db.dump() # save the db to file
File "/usr/local/lib/python3.10/dist-packages/pickledb.py", line 92, in dump
json.dump(self.db, open(self.loco, 'wt'))
FileNotFoundError: [Errno 2] No such file or directory: ''

in config.py I have the following line:

PICKLE_DB=""

FYI, it has same errors on running the RUN go install -... /nuclei@latest but I skipped nuclei installation.
Thank you!

error when installing naabu

I can't install naabu and I don't know what's missing:

 89 |     RUN apt-get install libpcap-dev -y
  90 | >>> RUN go get -v github.com/projectdiscovery/naabu/v2/cmd/naabu
  91 |     #RUN go install -v github.com/projectdiscovery/naabu/v2/cmd/naabu@latest > /tmp/naabu_install.log 2>&1
  92 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c go get -v github.com/projectdiscovery/naabu/v2/cmd/naabu" did not complete successfully: exit code: 1
Traceback (most recent call last):
  File "/usr/local/bin/lithops", line 8, in <module>
    sys.exit(lithops_cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lithops/scripts/cli.py", line 456, in build
    compute_handler.build_runtime(runtime_name, file, ctx.args)
  File "/usr/local/lib/python3.10/dist-packages/lithops/serverless/serverless.py", line 66, in build_runtime
    self.backend.build_runtime(runtime_name, file, extra_args)
  File "/usr/local/lib/python3.10/dist-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 354, in build_runtime
    utils.run_command(cmd)
  File "/usr/local/lib/python3.10/dist-packages/lithops/utils.py", line 713, in run_command
    sp.check_call(cmd.split())
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/docker', 'build', '-t', 'sc-runtime', '-f', 'Dockerfile', '.']' returned non-zero exit status 1.
2024-04-25 17:50:07,199 [INFO] config.py:134 -- Lithops v2.8.0
2024-04-25 17:50:07,200 [DEBUG] config.py:93 -- Loading configuration from /root/.lithops/config

runtime error

Hey I followed all the steps you mentioned in the wiki I also checked my aws account and the lambda function is created but I just keep getting the Could not execute the runtime error.

(env) ➜  ShadowClone git:(main) ✗ python shadowbrute.py -d netflix.com -w ~/tools/wordlist/best-dns-wordlist.txt
2023-05-02 01:13:42,902 [INFO] File with same name already exists in the bucket, skipping upload
2023-05-02 01:13:42,908 [INFO] config.py:134 -- Lithops v2.8.0
2023-05-02 01:13:42,929 [INFO] aws_s3.py:60 -- S3 client created - Region: ap-south-1
2023-05-02 01:13:44,078 [INFO] aws_lambda.py:94 -- AWS Lambda client created - Region: ap-south-1
2023-05-02 01:13:44,079 [INFO] invokers.py:108 -- ExecutorID ec7ffe-0 | JobID M000 - Selected Runtime: sc-runtime - 512MB
2023-05-02 01:13:44,079 [ERROR] Could not execute the runtime.

Nmap XML output?

Is there a way to output in xml format, or JSON in Httpx?

This tools rules!

Thanks

Custom command for each input line

I admire the speed of ShadowClone!

There is an use case where I want to run httpx on port http:9000 against a specified hosts, and on port https:8443 against a specified other hosts. Is this supported by ShadowClone?

use with azure

any docs to use with azure?

Rate limit error while running command

Any solution for this ?

(env) kkkk:ShadowClone:% python shadowclone.py -i subdomains.txt --split 300 -o httpxresults.txt -c "/go/bin/httpx -l {INPUT} -t 100"                                                                   <main ✗>
2022-10-20 04:52:29,805 [INFO] Splitting input file into chunks of 300 lines
2022-10-20 04:52:29,869 [INFO] Uploading chunks to storage
2022-10-20 04:52:29,886 [INFO] lithops.config -- Lithops v2.5.8
2022-10-20 04:52:29,928 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: us-east-1
2022-10-20 04:52:30,062 [INFO] lithops.serverless.backends.aws_lambda.aws_lambda -- AWS Lambda client created - Region: us-east-1
2022-10-20 04:52:30,064 [INFO] lithops.invokers -- ExecutorID 85b642-0 | JobID M000 - Selected Runtime: lithops_v2-5-8_7vdl/sc-runtime1 - 512MB
2022-10-20 04:52:37,246 [INFO] lithops.invokers -- ExecutorID 85b642-0 | JobID M000 - Starting function invocation: execute_command() - Total: 167 activations
exception calling callback for <Future at 0x7fedac13f9d0 state=finished raised Exception>
Traceback (most recent call last):
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 328, in _invoke_callbacks
    callback(self)
  File "/home/op/env/lib/python3.8/site-packages/lithops/invokers.py", line 439, in _callback
    future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/op/env/lib/python3.8/site-packages/lithops/invokers.py", line 370, in _invoke_task
    activation_id = self.compute_handler.invoke(payload)
  File "/home/op/env/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 59, in invoke
    return self.backend.invoke(runtime_name, runtime_memory, job_payload)
  File "/home/op/env/lib/python3.8/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 577, in invoke
    raise Exception('Error {}: {}'.format(r.status_code, r.text))
Exception: Error 429: {"Reason":"CallerRateLimitExceeded","Type":"User","message":"Rate exceeded"}
exception calling callback for <Future at 0x7fed5f95ceb0 state=finished raised Exception>
Traceback (most recent call last):
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 328, in _invoke_callbacks
    callback(self)
  File "/home/op/env/lib/python3.8/site-packages/lithops/invokers.py", line 439, in _callback
    future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/op/env/lib/python3.8/site-packages/lithops/invokers.py", line 370, in _invoke_task
    activation_id = self.compute_handler.invoke(payload)
  File "/home/op/env/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 59, in invoke
    return self.backend.invoke(runtime_name, runtime_memory, job_payload)
  File "/home/op/env/lib/python3.8/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 577, in invoke
    raise Exception('Error {}: {}'.format(r.status_code, r.text))
Exception: Error 429: {"Reason":"CallerRateLimitExceeded","Type":"User","message":"Rate exceeded"}
exception calling callback for <Future at 0x7fedac1ae2b0 state=finished raised Exception>

python version

hello,
when i try to run the tool i get the following error

2024-06-17 18:18:35,227 [ERROR] Could not execute the runtime.
The indicated runtime 'sc-runtime' is running Python 3.10 and it is not compatible with the local Python version 3.11

please assist.

nuclei and httpx while install through error

2024-06-03 09:59:55,308 [INFO] executors.py:612 -- ExecutorID f4e85c-0 - Cleaning temporary data
Encountered a bad command exit code!

Command: '/go/bin/ffuf -u /tmp/infile'

Exit code: 127

Stdout: already printed

Stderr: already printed

2024-06-03 09:55:53,328 [INFO] executors.py:612 -- ExecutorID ff142e-0 - Cleaning temporary data
Encountered a bad command exit code!

Command: 'cat /tmp/infile | /go/bin/ffuf'

Exit code: 127

Stdout: already printed

Stderr: already printed

Encountered a bad command exit code!

Command: 'cat /tmp/infile | /go/bin/ffuf'

Exit code: 127

Stdout: already printed

Stderr: already printed

Encountered a bad command exit code!

Command: 'cat /tmp/infile | /go/bin/ffuf'

Exit code: 127

Stdout: already printed

Stderr: already printed

nuclei installed but command working can you help out this?

Usage examples: Help needed

I've been trying to play around with shadowclones, but need help with commands for below situations:

Running a single nuclei template on the local machine for all the targets in resolved_subs.txt
Running multiple nuclei templates on local machine for a single url
Running subfinder on roots.txt with the providers-config.yaml containing all the api-keys.

@fyoorer, please help.

digitalocean support

hello. can u add digitalocean support ?

Running Nuclei

anyone who ran nuclei using this tool?
I'm getting the following error.

                    __     _
       ____  __  _______/ /__  (_)
      / __ \/ / / / ___/ / _ \/ /
     / / / / /_/ / /__/ /  __/ /
    /_/ /_/\__,_/\___/_/\___/_/   v2.9.2
    
    		projectdiscovery.io
    
    [INF] nuclei-templates are not installed, installing...
    [INF] nuclei-templates are not installed, installing...
    [FTL] Could not create runner: [:RUNTIME] could not create config file <- open /home/sbx_user1051/.config/nuclei/reporting-config.yaml: no such file or directory
    Error in running the command:/go/bin/nuclei -list /tmp/infile -es info
    ----------------------------------------------------------
    2023-05-02 00:44:27,639 [INFO] jobrunner.py:239 -- Success function execution
    2023-05-02 00:44:27,640 [INFO] jobrunner.py:311 -- Process finished
]

Activation: 'sc-runtime' (**********id**************)
[
    2023-05-02 00:44:26,303 [INFO] handler.py:152 -- Lithops v2.9.0 - Starting AWS Lambda execution
    2023-05-02 00:44:26,303 [INFO] handler.py:153 -- Execution ID: fbb4d8-0-M000/00003
    2023-05-02 00:44:26,322 [INFO] aws_s3.py:67 -- S3 client created - Region: ap-south-1
    2023-05-02 00:44:26,412 [INFO] jobrunner.py:134 -- Getting dataset from aws_s3://my-bucket
    2023-05-02 00:44:26,504 [INFO] jobrunner.py:180 -- Chunk: 1/1 - Size: 3464 - Range: 0-3463
    2023-05-02 00:44:26,505 [INFO] jobrunner.py:233 -- Going to execute 'execute_command()'
    ---------------------- FUNCTION LOG ----------------------
    [ERR] failed to create config directory at /home/sbx_user1051/.config/nuclei got: mkdir /home/sbx_user1051: read-only file system
    [ERR] failed to write config file at /home/sbx_user1051/.config/nuclei/.templates-config.json got: [:RUNTIME] could not create nuclei config directory at /home/sbx_user1051/.config/nuclei <- mkdir /home/sbx_user1051: read-only file system

I am using the following commands

python3 shadowclone.py -i ~/bbdata/alive.txt --split 200 -o nuke.txt -c "/go/bin/nuclei -list {INPUT} -es info

 python3 shadowclone.py -i ~/bbdata/alive.txt --split 200 -o nuke.txt -c "/go/bin/nuclei -list {INPUT} -t $HOME/nuclei-templates/ -es info"

But still getting the same error can anyone tell if there is anything wrong I'm doing here?

nmap and masscan usage examples

Hi there,
Can you help with nmap and masscan usage examples? I don't seem to get the right syntax to get an output.
Thanks in advance!

FAIL:vinternal_1: can't open adapter: libpcap not loaded - MASSCAN

After trying hard to install masscan, I'm still encountering the following error:

---------------------- FUNCTION LOG ----------------------
    [-] FAIL: failed to load libpcap shared library
        [hint]: you must install libpcap or WinPcap
    FAIL:vinternal_1: can't open adapter: libpcap not loaded
    [-] if:vinternal_1:init: failed
    Error in running the command:/masscan/bin/masscan -p 80 -iL /tmp/infile -oX masscan.xml --rate=1000
----------------------------------------------------------

I added libpcap-dev to the Dockerfile, has anyone already solved this problem?

ARM64

Any where to specify ARM64 in AWS Lambda via Lithops?

Aside from that, modifying the Dockerfile to --plathform=linux/amd64 doesn't seem to make it work on a M1 MacBook.

CallerRateLimitExceeded error

emergency exit to get output?

This is a weird reccomendation but what about a emergency exit to get output before getting runtime error?

Let's say we run a command that is important and we want to get the output, if it gets runtime error we can't see anything from the commands we just run.

It would be cool that if it keeps track of the runtime, a few seconds before getting to the error it saves the output from the command we just run.

Google Cloud Instructions

Is there a possibility for a Google cloud instruction by anyone that has attempted it.

[Documentation] Apple Silicon Users

Installation Notes for Passerby Apple Silicon Users

You may have noticed that the build breaks at lithops runtime build sc-runtime -f Dockerfile. Documenting this information here so that others can get it installed properly.

brew install [email protected]

Modify Dockerfile references:

FROM

to

FROM --platform=linux/amd64

Run the build:

lithops runtime build sc-runtime -f Dockerfile

Rate limit trigger

Any way to fix rate limit error,

Exception: Error 429: {"Reason":"CallerRateLimitExceeded","Type":"User","message":"Rate exceeded"}
2023-05-02 07:02:53,124 [INFO] wait.py:97 -- ExecutorID 4c24f3-0 - Getting results from 231 function activations
exception calling callback for <Future at 0x7ff08c10a580 state=finished raised Exception>
Traceback (most recent call last):
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 330, in _invoke_callbacks
    callback(self)
  File "/root/ShadowClone/env/lib/python3.9/site-packages/lithops/invokers.py", line 435, in _callback
    future.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/ShadowClone/env/lib/python3.9/site-packages/lithops/invokers.py", line 366, in _invoke_task
    activation_id = self.compute_handler.invoke(payload)
  File "/root/ShadowClone/env/lib/python3.9/site-packages/lithops/serverless/serverless.py", line 59, in invoke
    return self.backend.invoke(runtime_name, runtime_memory, job_payload)
  File "/root/ShadowClone/env/lib/python3.9/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 658, in invoke
    raise Exception('Error {}: {}'.format(r.status_code, r.text))
Exception: Error 429: {"Reason":"CallerRateLimitExceeded","Type":"User","message":"Rate exceeded"}

Upgrade Dockerfile to 1.18

The docker file is pulling 1.17 which is not compatible with httpx anymore.

Error coming while running Nuclei

Encountered a bad command exit code!

Command: 'nuclei -duc -l /tmp/infile -t /nuclei-templates -config-directory /tmp/'

Exit code: 127

Stdout: already printed

Stderr: already printed

getting error while running this command lithops runtime build sc-runtime -f Dockerfile

2022-01-24 14:04:46,069 [INFO] lithops.config -- Lithops v2.5.8
2022-01-24 14:04:46,069 [DEBUG] lithops.config -- Loading configuration from /root/.lithops/config
2022-01-24 14:04:46,139 [DEBUG] lithops.config -- Loading Serverless backend module: aws_lambda
2022-01-24 14:04:46,240 [DEBUG] lithops.serverless.backends.aws_lambda.aws_lambda -- Creating AWS Lambda client
2022-01-24 14:04:46,240 [DEBUG] lithops.serverless.backends.aws_lambda.aws_lambda -- Creating Boto3 AWS Session and Lambda Client
2022-01-24 14:04:47,588 [INFO] lithops.serverless.backends.aws_lambda.aws_lambda -- AWS Lambda client created - Region: us-east-1
2022-01-24 14:04:47,589 [INFO] lithops.serverless.backends.aws_lambda.aws_lambda -- Going to create runtime -d for AWS Lambda
2022-01-24 14:04:47,590 [DEBUG] lithops.utils -- Creating function handler zip in lithops_lambda.zip
Traceback (most recent call last):
File "/home/cherry/Desktop/ShadowClone/env/bin/lithops", line 8, in
sys.exit(lithops_cli())
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/lithops/scripts/cli.py", line 465, in build
compute_handler.build_runtime(name, file, ctx.args)
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 66, in build_runtime
self.backend.build_runtime(runtime_name, file, extra_args)
File "/home/cherry/Desktop/ShadowClone/env/lib/python3.8/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 319, in build_runtime
subprocess.check_call(cmd.split())
File "/usr/lib/python3.8/subprocess.py", line 359, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python3.8/subprocess.py", line 340, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.8/subprocess.py", line 858, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'None'

Could not execute the runtime

Using AWS Runtime.

python3 shadowbrute.py -d REDACTED -w subdomain_wordlist.txt
2022-01-24 16:47:39,414 [INFO] File with same name already exists in the bucket, skipping upload
2022-01-24 16:47:39,414 [ERROR] Could not execute the runtime.

Could not execute the runtime. How can I debug?

Question please

Please how can you set up with Azure ASAP.Also can I use additional tools with it and will it work like the pre-installed tools.

Issue: OSError: [Errno 24] Too many open files when running shadowclone.py

When running shadowclone.py with a large input file, the program fails with the following error message:

(env) ➜  ShadowClone git:(main) ✗ python shadowclone.py -i results_github.txt -s 1000 -o output_shadow -c "/go/bin/httpx -l {INPUT}"
2023-05-05 14:39:52,314 [INFO] Splitting input file into chunks of 1000 lines
2023-05-05 14:39:52,410 [INFO] Uploading chunks to storage
runtime-shadowclone/dbb8cfdc-92ed-4b8d-973e-d21edad30cd0
Traceback (most recent call last):
  File "/Users/hnguyen/arsenal/ShadowClone/shadowclone.py", line 174, in <module>
    filekeys = pool.map(upload_to_bucket, chunks)
  File "/Users/hnguyen/.pyenv/versions/3.9.5/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/hnguyen/.pyenv/versions/3.9.5/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/Users/hnguyen/.pyenv/versions/3.9.5/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/hnguyen/.pyenv/versions/3.9.5/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/Users/hnguyen/arsenal/ShadowClone/shadowclone.py", line 44, in upload_to_bucket
    chunk_hash = perform_hashing(chunk)
  File "/Users/hnguyen/arsenal/ShadowClone/hasher.py", line 8, in perform_hashing
    return hashlib.md5(file_as_bytes(open(fname, 'rb'))).hexdigest()
OSError: [Errno 24] Too many open files: '/var/folders/q2/v0knyw_n45598tj6xm9clf4r0000gp/T/small_file_100000.txt'

This error occurs because the program is trying to open too many files at once and has exceeded the system limit on the number of files that can be open simultaneously.

Steps to reproduce:

Run shadowclone.py with a large input file.
Wait for the program to fail with the above error message.
Expected behavior:
The program should be able to process large input files without exceeding the system limit on the number of open files.

Actual behavior:
The program fails with an OSError due to too many open files.

Proposed solution:
One possible solution to this issue is to modify the upload_to_bucket() function to use a with statement to automatically close the file after it has been read. This would prevent the number of open files from exceeding the system limit.

Note:
I have the fix and I could submit a pull request.

Anyone knows how to run X8 or Arjun with Shadowclone?

I have tested Shadowclone and it's working gret, I installed x8, however do not have any idea how I can run it with shadowclone. Normal command for x8 is:

x8 -u URL -X GET POST -w wordlist.txt

How to convert this for shadowclone?

Thanks

Issue after running "lithops runtime deploy sc-runtime --memory 512 --timeout 300"

2023-05-02 18:42:13,290 [DEBUG] aws_lambda.py:716 -- Extracting runtime metadata from: sc-runtime
Traceback (most recent call last):
File "/home/ubuntu/arsenal/ShadowClone/env/bin/lithops", line 8, in
sys.exit(lithops_cli())
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/lithops/scripts/cli.py", line 496, in deploy
runtime_meta = compute_handler.deploy_runtime(runtime_name, runtime_memory, runtime_timeout)
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/lithops/serverless/serverless.py", line 73, in deploy_runtime
return self.backend.deploy_runtime(runtime_name, memory, timeout=timeout)
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 538, in deploy_runtime
runtime_meta = self._generate_runtime_meta(runtime_name, memory)
File "/home/ubuntu/arsenal/ShadowClone/env/lib/python3.9/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 730, in _generate_runtime_meta
raise Exception('An error occurred: {}'.format(result))
Exception: An error occurred: {'errorType': 'Runtime.InvalidEntrypoint', 'errorMessage': 'RequestId: d71d7abf-1fa0-4509-a7ad-c83772585ae4 Error: fork/exec /usr/local/bin/python: exec format error'}

FFUF Example Usage?

Hi there,

Do you have an FFUF example usage? I kept getting errors parsing params for example:

python3 shadowclone.py -i input.txt -o output.txt -s 1 -c "cat {INPUT} | xargs -I @ -c '/go/bin/ffuf -u @ -w wordlist.txt -mc 200'"

Deleting files from bucket after run

I've noticed that billing has appeared in my AWS account, much quicker than I expected.
In my AWS account, I can see storage buckets containing files that may be large and never erased..
I saw that there is a function delete_bucket_files() in shadowclone.py to handle this:

but you comment the function call, is there a reason for that?

lithops create command does not exist.

Hello,

I've tried running the "lithops runtime create sc-runtime --memory 512 --timeout 300" command as instructed in the wiki, however it appears that that command does not exist. I suspect that I maybe running that command in the wrong place.

user@Administrators-MacBook-Pro-7 ~/arsenal/ShadowClone [16:48:40]
(recon-venv) > $ lithops runtime create sc-runtime --memory 512 --timeout 300 [±main ●●]
Usage: lithops runtime [OPTIONS] COMMAND [ARGS]...
Try 'lithops runtime --help' for help.

Error: No such command 'create'.

'Cannot allocate memory' when running tlsx 1.1.5

When trying to run tlsx 1.1.5 using shadowclone with an input file that contains 194359 lines, split into 1000 chunks of 195 lines, I receive the following error:

2023-11-29 13:31:00,025 [INFO] executors.py:612 -- ExecutorID 5072aa-0 - Cleaning temporary data
2023-11-29 13:31:00,037 [ERROR] Could not execute the runtime.
[Errno 12] Cannot allocate memory

The runtime I'm using has 512 MB allocated: sc-runtime4 - 512MB
When running it with a single line of input it seems to work fine - wonder if tlsx has a memory leak perhaps? Any suggestions to help narrow down the issue?

error output when running

Hello @fyoorer @vysecurity

I managed to configure this tool with the steps mentioned, but when running the output only shows $HOME is not defined. What really happened?

running findomain with shadowclone

hey, wanted to know if someone sucesfully run findomain with shadowclone.

if yes I would appreciate sharing the command in the dockerfile + the one for executing it <3. I seem to struggle to install it properly and getting:

Command: '/usr/local/bin/findomain -f /tmp/infile -q'

Exit code: 127

Stdout: already printed

Stderr: already printed

S3 Usage

Is it normal that the S3 usage is so high?

Exception: Error 429: {"Reason":"CallerRateLimitExceeded","Type":"User","message":"Rate exceeded"}`

exception calling callback for <Future at 0x7fd8fc0728e0 state=finished raised Exception> Traceback (most recent call last): File "/usr/lib/python3.9/concurrent/futures/_base.py", line 330, in _invoke_callbacks callback(self) File "/usr/local/lib/python3.9/dist-packages/lithops/invokers.py", line 435, in _callback future.result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 439, in result return self.__get_result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.9/dist-packages/lithops/invokers.py", line 366, in _invoke_task activation_id = self.compute_handler.invoke(payload) File "/usr/local/lib/python3.9/dist-packages/lithops/serverless/serverless.py", line 59, in invoke return self.backend.invoke(runtime_name, runtime_memory, job_payload) File "/usr/local/lib/python3.9/dist-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 658, in invoke raise Exception('Error {}: {}'.format(r.status_code, r.text)) Exception: Error 429: {"Reason":"CallerRateLimitExceeded","Type":"User","message":"Rate exceeded"}

Cannot install Nuclei from the Dockerfile

Whenever I run the command to build from the base Dockerfile from the repo, nuclei fails to install.

Here is the error I'm getting:

fyoorer / shadowclone Goto Github PK

shadowclone's Introduction

ShadowClone

Installation

Usage

How it works

Features

Comparison

Demo

References

Free Tiers

Similar Tools

Support

Disclaimer

shadowclone's People

Contributors

Stargazers

Watchers

Forkers

shadowclone's Issues

UPDATE: Fixed all errors Message me on Discord I'll send you starting to ending commands to properly install this tool, because official wiki is not updated. USERNAME: ba1van7

Installation Notes for Passerby Apple Silicon Users

Recommend Projects

Recommend Topics

Recommend Org