Code Monkey home page Code Monkey logo

steampipe-mod-aws-thrifty's Introduction

AWS Thrifty Mod for Powerpipe

Important

Powerpipe is now the preferred way to run this mod! Migrating from Steampipe →

All v0.x versions of this mod will work in both Steampipe and Powerpipe, but v1.0.0 onwards will be in Powerpipe format only.

An AWS cost savings and waste checking tool.

Run checks in a dashboard:

image

Or in a terminal:

image

Documentation

Getting Started

Installation

Install Powerpipe (https://powerpipe.io/downloads), or use Brew:

brew install turbot/tap/powerpipe

This mod also requires Steampipe with the AWS plugin as the data source. Install Steampipe (https://steampipe.io/downloads), or use Brew:

brew install turbot/tap/steampipe
steampipe plugin install aws

Steampipe will automatically use your default AWS credentials. Optionally, you can setup multiple accounts or customize AWS credentials.

Finally, install the mod:

mkdir dashboards
cd dashboards
powerpipe mod init
powerpipe mod install github.com/turbot/steampipe-mod-aws-thrifty

Browsing Dashboards

Start Steampipe as the data source:

steampipe service start

Start the dashboard server:

powerpipe server

Browse and view your dashboards at http://localhost:9033.

Running Checks in Your Terminal

Instead of running benchmarks in a dashboard, you can also run them within your terminal with the powerpipe benchmark command:

List available benchmarks:

powerpipe benchmark list

Run a benchmark:

powerpipe benchmark run ec2

Different output formats are also available, for more information please see Output Formats.

Configure Variables

Several benchmarks have input variables that can be configured to better match your environment and requirements. Each variable has a default defined in its source file, e.g., controls/rds.sp, but these can be overwritten in several ways:

It's easiest to setup your vars file, starting with the sample:

cp steampipe.spvars.example steampipe.spvars
vi steampipe.spvars

Alternatively you can pass variables on the command line:

powerpipe benchmark run ec2 --var=ec2_running_instance_age_max_days=90

Or through environment variables:

export PP_VAR_ec2_running_instance_age_max_days=90 
powerpipe control run long_running_ec2_instances

These are only some of the ways you can set variables. For a full list, please see Passing Input Variables.

Common and Tag Dimensions

The benchmark queries use common properties (like account_id, connection_name and region) and tags that are defined in the form of a default list of strings in the variables.sp file. These properties can be overwritten in several ways:

It's easiest to setup your vars file, starting with the sample:

cp steampipe.spvars.example steampipe.spvars
vi steampipe.spvars

Alternatively you can pass variables on the command line:

powerpipe benchmark run cloudfront --var 'tag_dimensions=["Environment", "Owner"]'

Or through environment variables:

export PP_VAR_common_dimensions='["account_id", "connection_name", "region"]'
export PP_VAR_tag_dimensions='["Environment", "Owner"]'
powerpipe benchmark run cloudfront

Open Source & Contributing

This repository is published under the Apache 2.0 license. Please see our code of conduct. We look forward to collaborating with you!

Steampipe and Powerpipe are products produced from this open source software, exclusively by Turbot HQ, Inc. They are distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get Involved

Join #powerpipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

steampipe-mod-aws-thrifty's People

Contributors

bigdatasourav avatar bluedoors avatar cbruno10 avatar dboeke avatar debabrat-git avatar e-gineer avatar joshrosen avatar khushboo9024 avatar madhushreeray30 avatar misraved avatar pranavpeshwe avatar priyanka-chatterjee-2000 avatar sankeyraut avatar shivani1982 avatar subhajit97 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

steampipe-mod-aws-thrifty's Issues

Unexpected AWS Cost Explorer Costs

Describe the bug
After starting to use this mod, I noticed that I'm seeing high usage of AWS Cost Explorer. After some digging, it appears that this module makes on average, 71 requests to the Cost Explorer API, which translates to $0.7/day. For production environments, this is a no-brainer but for testing, I thought that there should have been some type of warning as I left Steampipe running for over a month and got about $24 just from running this. I was running it on an AWS Organizations management account

Steampipe version (steampipe -v)
v0.13.3

Plugin version (steampipe plugin list)
v0.50.1

To reproduce

Expected behavior
It would be great if there was some type of warning related to this or some type of way to set this to only run once a day

Additional context

Add VPC control to check for VPCs with no ENIs

Is your feature request related to a problem? Please describe.
I'd like to be able to check for VPCs with no ENIs in them so I can clean them up.

Describe the solution you'd like
Add a control that checks for VPCs with no ENI.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Cloudwatch Throttle Exception

Describe the bug
On the check called Unused log streams should be removed if not required, I receive the error:

operation error CloudWatch Logs: DescribeLogStreams, exceeded maximum number of attempts, 9, https response error StatusCode: 400, RequestID: 8fd06b8d-5f20-403b-a2a3-2ff825760378, api error ThrottlingException: Rate exceeded (SQLSTATE HV000)

Steampipe version (steampipe -v)
Example: v0.17.4

Plugin version (steampipe plugin list)
[email protected]

To reproduce
Using a connection to an AWS account with a lot of Cloudwatch log groups (my account has 320) , click on the dashboard and wait for it to fail with the above error

Expected behavior
I expect it to at least return what it has gathered and better yet would be to return what it has and then continue gathering after waiting

Additional context
Add any other context about the problem here.

Add additional controls for AWS Thrifty

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add a control that checks if auto scaling is enabled for DynamoDB table

Is your feature request related to a problem? Please describe.
Add a control that checks if autoscaling is enabled for the DynamoDB table. This is essential for cost control.

Describe the solution you'd like
Use the dynamodb table queries to check if autoscaling is enabled or not.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

vpc_nat_gateway_unused matches every nat gateway

This query tries to find unused nat gateways. In words:

That's imo not correct for multiple reasons:

  1. the subnet_id attached to a NAT gateway is always the public one exclusive attached to the gateway (isn't it?). This means this query will never find any nat.subnet_id = i.subnet_id
  2. It's not only ec2 using VPC´s and NAT gateways...what about lambda, etc?

A solution could be to find NAT gateways without transfer costs, but running hours?

Add control that checks VPC flow log data for unused resources

Is your feature request related to a problem? Please describe.
I'd like to be able to look for unused/underutilized resources by looking at VPC flow log data and checking which resources have no traffic in/out to them.

Describe the solution you'd like
Add a control that checks VPC flow log data and alarms on resources with no inbound/outbound traffic.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add controls for all supported resources that check if cost allocation tags are created

Is your feature request related to a problem? Please describe.
I'd like to be able to see resources have cost allocation tags on them or not; if not, alarm for those resources.

Describe the solution you'd like
Add controls that check for cost allocation tags.

Describe alternatives you've considered
N/A

Additional context
Add any other context or screenshots about the feature request here.

Transit Gateway Data Processing Charges

When you centralize NAT gateway using Transit Gateway, you pay an extra Transit Gateway data processing charge — compared to the decentralized approach of running a NAT gateway in every VPC. In some edge cases when you send huge amounts of data through NAT gateway from a VPC, keeping the NAT local in the VPC to avoid the Transit Gateway data processing charge might be a more cost-effective option.

Refer to AWS whitepaper.

It would be nice to surface a control to track this if possible.

Update title, description, query names of thrifty controls.

Is your feature request related to a problem? Please describe.

  1. The title can be generic as in other Thrifty mods.
  2. Re-validate the status to show error
  3. Name of the queries should follow standard
+---------------------------------------------------------------------------------+--------+--------------------------------
| resource                                                                        | status | reason                                                                 | region    | account_id   |
+---------------------------------------------------------------------------------+--------+------------------------------------------------------------------------+-----------+--------------+
| arn:aws:lambda:us-east-1:533123462495:function:SecretsManagermysqldbsecret      | error  | CloudWatch metrics not available for SecretsManagermysqldbsecret.      | us-east-1 | 533123462495 |
| arn:aws:lambda:us-east-1:533123462495:function:SecretsManagermysqldb-admin-cred | error  | CloudWatch metrics not available for SecretsManagermysqldb-admin-cred. | us-east-1 | 533123462495 |
+---------------------------------------------------------------------------------+--------+------------------------------------------------------------------------+-----------+--------------+

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Refactor control "cw_log_stream_unused"

Describe the bug
This control breaks when executed against the account(s) with a large no of data.

Unused log streams should be removed if not required .................................................................................................................. 26,017 / 26,226 [==========]
|   
|   ERROR: timeout: context deadline exceeded

Query

    select
      arn as resource,
      case
        when last_ingestion_time is null then 'error'
        when date_part('day', now() - last_ingestion_time) > 90 then 'alarm'
        else 'ok'
      end as status,
      case
        when last_ingestion_time is null then name || ' is not reporting a last ingestion time.'
        else name || ' last log ingestion was ' || date_part('day', now() - last_ingestion_time) || ' days ago.'
      end as reason
    from
      aws_cloudwatch_log_stream;

Steampipe version (steampipe -v)
Example: v0.18.5

Plugin version (steampipe plugin list)
Example: v0.92.2

To reproduce
Steps to reproduce the behavior (please include relevant code and/or commands).

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Cost for underutilised resources

Is your feature request related to a problem? Please describe.
The thrify mod shows list of underutilised resources which is a good starting point. However it will be good if we can show $ cost associated with the underutilised resource so that we can plan and prioritize action.

Describe the solution you'd like
Along with the resource list, please include the ondemand cost of the resource.

Describe alternatives you've considered
Recently a new table has been added to query the cost of the resource. (turbot/steampipe-plugin-aws#632) called as aws_pricing_product. Alternative is to join the query used in thrify mod with this table to come up with the cost manually.

Additional context
This will help create actionable dashboards with $ numbers and can help service owner set the priority for maximum cost savings.

Add/Update common dimension

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add additional thrifty control to check unused NAT gateways

Is your feature request related to a problem? Please describe.
NAT Gateway is charged on an hourly basis once it is provisioned and available. Hence, unused NAT gateways should be reviewed.

Describe the solution you'd like
Add a check to find out unused NAT gateways present in an AWS account.

Describe alternatives you've considered
N/A

Additional context
N/A

Add unused Network Firewall with NAT Gateway check

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Reference https://aws.amazon.com/network-firewall/pricing/

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add default values to all variables to better support `mod get` functionality

When getting a mod with variables through mod get, if there are no defaults defined and no values defined locally, any steampipe command will then prompt the user for those variables. We should add defaults and provide instructions with a sample Steampipe variable file for a smoother mod get experience and local experience as well.

Add additional controls for AWS Thrifty

Is your feature request related to a problem? Please describe.
Add new additional controls for AWS Thrifty such as ECS, EMR and Redshift

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Update 'low_utilization_ec2_instance.sql` query to check for max(average) instead of avg(max) utilization

Is your feature request related to a problem? Please describe.
This query is not as accurate as it could be due to us checking the max(average) instead of avg(max).

Describe the solution you'd like
Update the query to check for max(average) instead of avg(max).

Describe alternatives you've considered
N/A

Additional context
Add any other context or screenshots about the feature request here.

Fix broken link in Lambda checks

Describe the bug

  1. https://hub.steampipe.io/mods/turbot/aws_thrifty/queries/lambda_function_timeout_7_days is broken.
  2. Improve descriptions

Steampipe version (steampipe -v)
Example: v0.3.0

Plugin version (steampipe plugin list)
Example: v0.5.0

To reproduce
Steps to reproduce the behavior (please include relevant code and/or commands).

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Add S3 bucket control that looks for development buckets with versioning enabled

Is your feature request related to a problem? Please describe.
I'd like to be able to check for development S3 buckets that have versioning enabled, which may be unnecessary.

Describe the solution you'd like
Add a control that alarms on development environment buckets with versioning enabled.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add query for thrifty

Is your feature request related to a problem? Please describe.
Add new queries

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

quoted param is not used by control

In old_ebs_snapshots.sql the param is quoted:

when start_time > current_timestamp - interval '$1 days' then 'ok'

As such it's just a string, so the query is incorrect.

per @johnsmyth, this variant works:

when start_time > current_timestamp - ($1 || ' days')::interval then 'ok'

We should probably search for this pattern across the corpus, a quick spot-check found this in gcp-thrifty:

current_date - interval '$1' day

False positive - redundant cloudtrails

Describe the bug
Thrifty Mod returns incorrect number of Global Trails when Org Trails are in use

FooliOrgTrails-ModifyEventsCloudTrail-xJdAZqCR4abt is one of 18 global trails.

Steampipe version (steampipe -v)
SPC v0.18.0

Plugin version (steampipe plugin list)
Plugin 0.18.0

To reproduce
In an AWS Org with 6 accounts and 3 CloudTrails defined by the organizational parent, the incorrect number of redundant trails is reported.

Create an AWS Org
Create a child account
Create three trails
Note that the number of trails is 6 not three

Expected behavior
The thrifty mod should not double count identical cloudtrails that are managed by the AWS Organizations management account.

Additional context
May be related to turbot/steampipe-mod-aws-compliance#536

Add EC2 instance controls that check for use of t2, m3 and m4 instances types

Is your feature request related to a problem? Please describe.
I'd like to be able to see if any EC2 instances use older generation t2, m3, and m4 instance types (t3 and m5 are more cost effective).

Describe the solution you'd like
Add a control that checks if instances are using the older instance types.

Describe alternatives you've considered
N/A

Additional context
Add any other context or screenshots about the feature request here.

Add control to check ElastiCache Redis Cluster Engine CPU utilization

Is your feature request related to a problem? Please describe.
Add control to check if ElastiCache Redis Cluster Engine CPU is underutilized.
https://aws.amazon.com/blogs/database/monitoring-best-practices-with-amazon-elasticache-for-redis-using-amazon-cloudwatch/
Describe the solution you'd like
Need to fetch information from aws_elasticache_redis_metric_engine_cpu_utilization_daily table

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Spelling correction for the query `s3_bucket_without_lifecycle`

Describe the bug
it should be cycle instead of cyle

Steampipe version (steampipe -v)
Example: v0.3.0

Plugin version (steampipe plugin list)
Example: v0.5.0

To reproduce
Steps to reproduce the behavior (please include relevant code and/or commands).

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Remove comments from the inline query of `vpc_nat_gateway_unused` control

Describe the bug
Because of the comments -- Required Columns and -- Additional Dimensions in vpc_nat_gateway_unused control's inline query, the output format of the query on the hub is broken.

https://hub.steampipe.io/mods/turbot/aws_thrifty/controls/control.vpc_nat_gateway_unused?context=benchmark.network#sql

Steampipe version (steampipe -v)
Example: v0.20.0-alpha.1

Plugin version (steampipe plugin list)
Example: v0.5.0

To reproduce
https://hub.steampipe.io/mods/turbot/aws_thrifty/controls/control.vpc_nat_gateway_unused?context=benchmark.network#sql

Expected behavior
The query should adhere to https://hub.steampipe.io/mods/turbot/aws_thrifty/controls/control.vpc_nat_gateway_unused?context=benchmark.network#sql format

Additional context
Add any other context about the problem here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.