johannesebke / aws_list_all Goto Github PK
View Code? Open in Web Editor NEWList all your AWS resources, all regions, all services.
License: MIT License
List all your AWS resources, all regions, all services.
License: MIT License
Let me know if you would like a PR for above.
I'd like to simply get a list of all resources in my account, giving their ARN. As a JSON query this is non trivial and probably impossible to get right since AWS uses many different ways of giving the ARN (example, sometimes it's just a parameter on the resource ( ARN: "value"), sometimes it's a differently named parameter ( "DBClusterARN") and sometimes
I have a workaround; change into the json directory and run
cat *.json | tr '"' '\n' | grep '^arn:aws' | sort | sed 's/:\*$//' | uniq
I don't see this as fully reliable though, for example if there are references to ARNs in a different account they will turn up which may or may not be a good thing, if there are references to old ARNs that no longer exist, the same.
I think the fix is to have this as an option to aws_list_all. This could start by gathering the values of all parameters which match ^(ARN|[^ ]*Arn)
and then printing out the unique values but if AWS changed something might need updated.
This would be deadly useful for those of us who do pci compliance
It should be possible to use the new AWS Cloud Control API, both to simplify the methods used to find resources, and to find resources that aren't currently found (and potentially to make it so that new resource types work without any code changes).
When trying a simple query:
aws-list-all query
the following error is generated:
~\envs\aws\lib\site-packages\aws_list_all\introspection.py", line 343, in get_regions_for_service
regions = set(get_service_regions()[requested_service])
KeyError: 'application-insights'
Dos this mean that the application-insight services reported by AWS is not available in any region? Although this doesnt make sense I would expect this type of error to handled rather than a crash.
OR
What am I doing wrong?
Thanks for your help.
Currently it only works with the [default] profile credentials. Please provide support for using a different profile from ~/.aws/credentials.
In general we have multiple profiles for different accounts for different purposes.
Is there a recommendation for a policy with the least rights that can be used to successfully run the script ?
I really like this project, its good to do some specific queries when one is interested in dig deeper. But It can be easily automated to extract all resources which are running in AWS and eventually draw some infra diagram. Is there any tool which is compatible with output of this tool to visualize infra? I am not ask to develop it here, its more if someone is aware of such app.
Thanks.
aws-list-all query --service ec2 --operation DescribeVpcs --directory data --verbose
Building set of queries to execute...
Service: ec2 | Region: us-west-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-northeast-2 | Operation: DescribeVpcs
Service: ec2 | Region: us-west-2 | Operation: DescribeVpcs
Service: ec2 | Region: ca-central-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-south-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-southeast-1 | Operation: DescribeVpcs
Service: ec2 | Region: sa-east-1 | Operation: DescribeVpcs
Service: ec2 | Region: us-east-2 | Operation: DescribeVpcs
Service: ec2 | Region: eu-west-1 | Operation: DescribeVpcs
Service: ec2 | Region: us-east-1 | Operation: DescribeVpcs
Service: ec2 | Region: eu-central-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-southeast-2 | Operation: DescribeVpcs
Service: ec2 | Region: eu-west-2 | Operation: DescribeVpcs
Service: ec2 | Region: eu-west-3 | Operation: DescribeVpcs
Service: ec2 | Region: eu-north-1 | Operation: DescribeVpcs
Service: ec2 | Region: ap-northeast-1 | Operation: DescribeVpcs
...done. Executing queries...
The task "...done. Executing queries..." gets in loop and never finish, any sugestion?
Thanks in advance
Hi I am new as well, thank you for creating this!
I am trying to use your module but am having trouble using it inside my python script... I need to list all services for each of my aws accounts but I don't know how I can use this module to do so.. I understand that it says it creates a json of data when running query/commands in the terminal but I cannot find any json produced. Please help/redirect thank you!
If run with a nlimit of 1024, aws-list-all runs into OSErrors due to too many open files (connections).
I cannot run the "quick start" in the README.
$ aws-list-all --region eu-west-1 --service ec2 --directory ./data/
usage: aws_list_all [-h] COMMAND ...
aws_list_all: error: argument COMMAND: invalid choice: 'eu-west-1' (choose from 'query', 'show', 'introspect', 'recreate-caches')
Is this a regression, or is the README out of date?
One apparent omission seems to be route53. It would be useful to get a complete list of domains.
Hi, Very nice work!
I tried aws_list_all in python 2.7 and it worked as expected without any problems. However when I moved to another system with a virtualenv using python 3.4, I get errors like the following.
--------------------------------------------------------------------------------
WARNING!
Your system limits the number of open files and network connections to 4096.
This may lead to failures during querying.
Please increase the hard limit of open files to at least 6000.
The configuration for hard limits is often found in /etc/security/limits.conf
--------------------------------------------------------------------------------
Increasing the open connection limit "nofile" from 3072 to 4096.
Building set of queries to execute...
Traceback (most recent call last):
File "/home/user/scripts/aws/lib/python3.4/site-packages/app_json_file_cache/function_cache.py", line 17, in f
return self.get(key)
File "/home/user/scripts/aws/lib/python3.4/site-packages/app_json_file_cache/data_cache.py", line 27, in get
return self._data[key_string]['data']
KeyError: '{"args": [], "kwargs": {}}'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/scripts/aws/bin/aws-list-all", line 10, in <module>
sys.exit(main())
File "/home/user/scripts/aws/lib/python3.4/site-packages/aws_list_all/__main__.py", line 142, in main
do_query(services, args.region, args.operation, verbose=args.verbose or 0, parallel=args.parallel)
File "/home/user/scripts/aws/lib/python3.4/site-packages/aws_list_all/query.py", line 203, in do_query
for region in get_regions_for_service(service, selected_regions):
File "/home/user/scripts/aws/lib/python3.4/site-packages/aws_list_all/introspection.py", line 373, in get_regions_for_service
regions = set(get_service_regions().get(requested_service, []))
File "/home/user/scripts/aws/lib/python3.4/site-packages/app_json_file_cache/function_cache.py", line 20, in f
return self._cheap_default_func(*args, **kwargs)
File "/home/user/scripts/aws/lib/python3.4/site-packages/aws_list_all/introspection.py", line 355, in packaged_service_regions
return load(resource_stream(__package__, 'service_regions.json'))['data']
File "/usr/lib64/python3.4/json/__init__.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.4/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
I will try troubleshoot later and come back in case I find something.
Is it possible to list all the route53 domains either based on the ResourceRecordSetCount in (route53_ListHostedZonesByName*) or any other possible methods ?
Hi Johannes,
Sorry to bother you, but I am curious how you generated the list of service and regions. I found the the links below in google, but It's a bit of challenging to scrape the data from them.
Do you have any secret souce?
It would be nice if the us-gov-west-1 and us-gov-east-1 regions were supported.
When running a query against a gov region:
$ aws-list-all query --directory aws_inventory --region us-gov-west-1 --service ec2 -v -v
Increasing the open connection limit "nofile" from 256 to 6000.
Building set of queries to execute...
...done. Executing queries...
...done
Expected results would be queries running against the region instead of no queries ran.
Are Regions and Services still up to date?
When I run the example (Ubuntu 18, python 3.6) it just hangs forever. Whenever I ctrl-c the process, it shows the same output:
me:~ $ python3 -m aws_list_all query --service ec2 --region us-east-1
^CTraceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 746, in next
item = self._items.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/dist-packages/aws_list_all/__main__.py", line 109, in <module>
main()
File "/usr/local/lib/python3.6/dist-packages/aws_list_all/__main__.py", line 87, in main
do_query(services, args.region, args.operation)
File "/usr/local/lib/python3.6/dist-packages/aws_list_all/query.py", line 91, in do_query
for result in ThreadPool(32).imap_unordered(acquire_listing, to_run):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 750, in next
self._cond.wait(timeout)
File "/usr/lib/python3.6/threading.py", line 295, in wait
waiter.acquire()
KeyboardInterrupt
me:~ $
Direct aws CLI commands like aws s3 ls
work fine.
Currently, if a nextToken is found in a response, we mark it as "non-complete" and show "> N" resources. In some cases (inspector) this leads to "> 0" listings. While this is correct, it would be nice to have an (optional) "complete mode" that follows up all nextTokens.
Hello,
I have a few issues:
aws-list-all --region eu-west-1 --service s3 --directory ./data/
Returns:
usage: aws_list_all [-h] COMMAND ... aws_list_all: error: argument COMMAND: invalid choice: 'eu-west-1' (choose from 'query', 'show', 'introspect', 'recreate-caches')
is it a personal or general problem?
my goal here is to retrieve information about the "internet gateway" service
Error:
cloudformation eu-north-1 ListStackSets ClientError('An error occurred (ValidationError) when calling the ListStackSets operation: AWS CloudFormation StackSets is not supported in eu-north-1')
Traceback (most recent call last):
File "/usr/local/bin/aws-list-all", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/aws_list_all/__main__.py", line 142, in main
do_query(services, args.region, args.operation, verbose=args.verbose or 0, parallel=args.parallel)
File "/usr/local/lib/python3.7/site-packages/aws_list_all/query.py", line 206, in do_query
print('Service: {: <28} | Region: {:<15} | Operation: {}'.format(service, region, operation))
TypeError: unsupported format string passed to NoneType.__format__
Newbie alert! Just getting into both AWS and Python.
Trying to run aws-list-all and I get the following error message.
Python 3.8.2
Windows 10.
(aws-list-all) C:\Program Files (x86)\Python38-32\aws-list-all\Scripts>aws-list-all query --region eu-west-1 --service ec2
Traceback (most recent call last):
File "c:\program files (x86)\python38-32\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\program files (x86)\python38-32\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Program Files (x86)\Python38-32\aws-list-all\Scripts\aws-list-all.exe\__main__.py", line 4, in <module>
File "c:\program files (x86)\python38-32\aws-list-all\lib\site-packages\aws_list_all\__main__.py", line 5, in <module>
from resource import getrlimit, setrlimit, RLIMIT_NOFILE
ModuleNotFoundError: No module named 'resource'
any tips on converting the json files to Excel? Tried pandas library, however the converted Excel files do not display the returned values in the right order of columns/rows.
I cannot execute any query against any service. This is always what happens:
aws-list-all query --region eu-west-1 --service ec2 --operation DescribeVpcs --verbose
Building set of queries to execute...
Service: ec2 | Region: eu-west-1 | Operation: DescribeVpcs
...done. Executing queries...
Regardless of operation or service, it just hangs. I've let it run for hours with no progress.
Determining if a key is AWS-managed would require a second call per key.
The Braket SearchQuantumTasks and SearchDevices APIs are such that they require a list as "filters" argument, yet this list can be zero-length. Therefore, there is no equivalent "ListQuantumTask" as in other services.
Relevant API documentation:
https://docs.aws.amazon.com/braket/latest/APIReference/API_SearchQuantumTasks.html
https://docs.aws.amazon.com/braket/latest/APIReference/API_SearchDevices.html
Example code illustrating the problem even without any Quantum Tasks:
boto3.client("braket", region_name="us-east-1").search_devices(filters=[])
Hi, trying to run the command to show all resources but the script goes out of memory.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
[...]
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
MemoryError
How much memory is required to "query" everything and how it could be optimized? Thanks a lot.
I tried to list my AWS resources in us-east-2
and got a KeyError
for at least one of the services:
$ aws-list-all query --region us-east-2 --directory ./data/
Building set of queries to execute...
Traceback (most recent call last):
File "/home/ubuntu/.local/share/virtualenvs/acm-api-UVOmdwT4/bin/aws-list-all", line 10, in <module>
sys.exit(main())
File "/home/ubuntu/.local/share/virtualenvs/acm-api-UVOmdwT4/lib/python3.7/site-packages/aws_list_all/__main__.py", line 115, in main
do_query(services, args.region, args.operation, verbose=args.verbose or 0)
File "/home/ubuntu/.local/share/virtualenvs/acm-api-UVOmdwT4/lib/python3.7/site-packages/aws_list_all/query.py", line 195, in do_query
for region in get_regions_for_service(service, selected_regions):
File "/home/ubuntu/.local/share/virtualenvs/acm-api-UVOmdwT4/lib/python3.7/site-packages/aws_list_all/introspection.py", line 343, in get_re
gions_for_service
regions = set(get_service_regions()[requested_service])
KeyError: 'managedblockchain'
I expect this is a service that does not exist in that region.
No response for s3, route53 etc.
https://travis-ci.org/github/JohannesEbke/aws_list_all/jobs/737406268
___________________________ test_get_service_regions ___________________________
def test_get_service_regions():
services = get_services()
regions = get_service_regions()
> assert set(services) - set(regions) == set()
E AssertionError: assert {'s3outposts'...stream-write'} == set()
E Extra items in the left set:
E 'timestream-query'
E 'timestream-write'
E 's3outposts'
E Use -v to get the full diff
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.