Code Monkey home page Code Monkey logo

folding-aws's Introduction

Welcome

Hi, I’m Sathyajith Bhat. I am an experienced Site Reliability Engineer (SRE) with over 15 years of experience in DevOps, Site Reliability Engineering, System Architecture, Performance Tuning, Infrastructure setup, DBA, Data Conversion and Observability.

I'm not fanatic about specific tools or services and believe in making the most available out of the tools that I get to work with. I have significant experience in the below services, tools and technologies:

Cloud Platforms

  • Amazon Web Services (core compute, networking, database and storage services)
  • Azure (core compute, networking, database and storage services)
  • Alibaba Cloud (core compute, networking, database and storage services)

Containers & Orchestration

  • Docker
  • Kubernetes

Infrastructure as Code

  • Terraform
  • AWS CDK

Configuration Management

  • Ansible
  • Chef

Observability

  • Prometheus
  • New Relic
  • Splunk
  • Datadog

I am the author of Practical Docker with Python (first & second edition) and co-author of The CDK Book. I work with a lot of different communities and have been recognized as an AWS Community Hero.

I love organizing and running events (physical as well as virtual) and have been part of organizing committee of events such as Barcamp Bangalore, AWS Community Day - Bangalore, India, South Asia, and CDK Day. In my free time I love playing games on my PC and/or Switch (typically ARPGs/City Builders/Simulators) and occasionally livestream gaming and coding on Twitch.

Projects I maintain

  • Spotify-DL - Downloads songs from any Spotify playlist.
  • Folding at Home - AWS - Bring up a complete AWS Compute stack with VPC, EC2, and other dependencies using AWS CDK.
  • CDK CDN - a CDK construct to deploy a CDN on AWS using S3 and CloudFront.
  • DevOps Workflow - Repo for demonstrating a typical devops workflow using Ansible, Terraform, Packer and Jenkins. A typical devops workflow consists of build -> test -> validate -> release cycle.
  • Source code for Practical Docker with Python - Source code for projects in Practical Docker with Python book.
  • The CDK Book - Examples and feedback on the CDK Book.

Reach out to me!

  • 💬 Ask me about AWS, Cloud Computing, Containers, DevOps
  • 📫 How to reach me: Twitter, LinkedIn, email
  • ⚡ Fun fact: I'm an avid gamer(RPGs, City Builders) and Community Moderator on Super User

folding-aws's People

Contributors

ninadsp avatar sathyabhat avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

ninadsp mdfranz

folding-aws's Issues

Make it possible to bring up a generic EC2 stack

#2 makes it possible to customize the stack. We can extend this to make the stack furthermore customizable and generic by making it possible to assign a name that will be used with the stack

Current config is so:

aws:
  # What AWS region do we use for VPC and ASG stacks?
  # optional
  region: us-east-1
  # What is the AMI ID of the Folding @ Home Image
  # mandatory
  ami_id: ami-1234567890
  # What ssh key is allowed access to the instances in the ASG?
  # mandatory
  ssh_key: super-private-ssh-key
  # What is the maximum price in USD that we're willing to pay?
  # mandatory
  max_spot_price: "1.234"
  # What IP addresses to allow security group access from?
  # mandatory
  ssh_allow_ip_range: 192.168.0.1/32
  # What instance type/sizing would you prefer?
  # optional
  ec2_instance_type: "c5n.large"
  # How many instances do you wish to use?
  # optional
  asg_size: 2

by adding an indent and another key which becomes a name, we can make it possible to bring up any asg with predetermined AMI, making it easier to bring up an EC2 ASG

aws:
  example-name:
    # What AWS region do we use for VPC and ASG stacks?
    # optional
    region: us-east-1
    # What is the AMI ID of the Folding @ Home Image
    # mandatory
    ami_id: ami-1234567890
    # What ssh key is allowed access to the instances in the ASG?
    # mandatory
    ssh_key: super-private-ssh-key
    # What is the maximum price in USD that we're willing to pay?
    # mandatory
    max_spot_price: "1.234"
    # What IP addresses to allow security group access from?
    # mandatory
    ssh_allow_ip_range: 192.168.0.1/32
    # What instance type/sizing would you prefer?
    # optional
    ec2_instance_type: "c5n.large"
    # How many instances do you wish to use?
    # optional
    asg_size: 2

If force flag is not set and spot price set to "" in the config cdk crashes

With the spot price not set in config, and force flag not passed, cdk crashes with EOF error.

sample config:

folding:
  # What AWS region do we use for VPC and ASG stacks?
  # optional
  region: eu-west-1
  # CIDR for the new vpc to be created
  # mandatory
  cidr: 10.0.0.0/16
  # What is the AMI ID of the Folding @ Home Image
  # mandatory
  ami_id: ami-0a8341e7c0a146e7f
  # What ssh key is allowed access to the instances in the ASG?
  # mandatory
  ssh_key: sathya_folding_ssh
  # What is the maximum price in USD that we're willing to pay?
  # If you leave this empty, we will fetch the current spot price
  # and raise the bid by a few cents. The instances will be set up
  # only after confirmation
  # optional
  max_spot_price: ""
  # What IP addresses to allow security group access from?
  # mandatory
  ssh_allow_ip_range: 84.232.204.212/32
  # What instance type/sizing would you prefer?
  # optional
  ec2_instance_type: "c5n.large"
  # How many instances do you wish to use?
  # optional
  asg_size: 2
  tags:
    - name: used_for
      value: folding

Behaviour:

cdk diff -c stack_name=folding --profile sathya                                            ✘ 1 
Got spot price: 0.042000
Stack folding-vpc
There were no differences
Stack folding-asg
Security Group Changes
┌───┬──────────────────────┬─────┬──────────┬───────────────────┐
│   │ Group                │ Dir │ Protocol │ Peer              │
├───┼──────────────────────┼─────┼──────────┼───────────────────┤
│ - │ ${allow-ssh.GroupId} │ In  │ TCP 22   │ 188.26.173.155/32 │
├───┼──────────────────────┼─────┼──────────┼───────────────────┤
│ + │ ${allow-ssh.GroupId} │ In  │ TCP 22   │ 84.232.204.212/32 │
└───┴──────────────────────┴─────┴──────────┴───────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Resources
[~] AWS::EC2::SecurityGroup allow-ssh allowsshD2EB9346 
 └─ [~] SecurityGroupIngress
     └─ @@ -1,7 +1,7 @@
        [ ] [
        [ ]   {
        [-]     "CidrIp": "188.26.173.155/32",
        [-]     "Description": "from 188.26.173.155/32:22",
        [+]     "CidrIp": "84.232.204.212/32",
        [+]     "Description": "from 84.232.204.212/32:22",
        [ ]     "FromPort": 22,
        [ ]     "IpProtocol": "tcp",
        [ ]     "ToPort": 22

(cdk) ~/c/folding-aws ❯❯❯ cdk diff -c stack_name=folding --profile sathya
Got spot price: 0.042000
You have not specified a maximum price to pay for the Spot Instances.
Are you willing to pay 0.062 US Dollars / hour for each instance? Note: You have requested for 2 instances of type c5n.large.
y/N: Traceback (most recent call last):
  File "app.py", line 85, in <module>
    cdk_init(stack_name, force_spot_price)
  File "app.py", line 50, in cdk_init
    asg_stack = AsgStack(app, 
  File "/home/sathyabhat/venv/cdk/lib/python3.8/site-packages/jsii/_runtime.py", line 66, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/sathyabhat/code/folding-aws/aws_stack/asg_stack.py", line 46, in __init__
    user_consent = input("y/N: ")
EOFError: EOF when reading a line
Subprocess exited with error 1

Allow some parameters to be specified to the stack

For other people to consume this project, it makes sense to convert the following into parameters that can be specified to the cdk app by the user:

  • AMI ID for the F@H VM
  • Region to set up the VPC and ASG stacks in
  • name of the SSH key
  • Maximum acceptable spot price
  • IP address that is allowed SSH access to the instances in the ASG

Add support for tags

Currently, no tags are set making cost analysis and asset tracking difficult. Tag support should be added.

We can make use of the config file to read the tags and apply them

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.