open-guides / og-aws Goto Github PK

General suggestions, and where to get help.

Good initial overview
http://www.slideshare.net/AmazonWebServices/encryption-and-key-management-in-aws

Compare/contrast with alternatives, including HSM, esp slide 39 there.

Follow-on for issues discussed #38

Bunch of tools related to this that we should sort though. Cover basic use cases and best tools and problems.

• Overall notes on when to use EFS vs EBS vs S3

Tools:

• ObjectiveFS (homepage (https://objectivefs.com/), HN (https://news.ycombinator.com/item?id=10117506))
  • Live FS (no disconnected usage), does support concurrent clients
• https://github.com/s3ql/s3ql (HN (https://news.ycombinator.com/item?id=10150684))
  • No concurrent clients?
• https://github.com/s3fs-fuse/s3fs-fuse
  • Doesn't do things like atomic rename of directories, mutual exclusion of open exclusive, append to file requires rewriting the whole file, no hard links, which can break the expectations of some programs.
• https://github.com/skoobe/riofs
• https://github.com/kahing/goofys

https://github.com/cloudfoundry
https://pivotal.io/platform

http://cloudacademy.com/blog/top-13-amazon-virtual-private-cloud-best-practices/

Another tip: Be careful not to pick a local network IP range that conflicts with the AWS VPC range.

More links like these:
http://royal.pingdom.com/2011/04/27/the-amazon-ec2-outage-time-to-learn-from-it/

For each covered service link to sections in awesome-aws, e.g. https://github.com/donnemartin/awesome-aws#dynamodb

Summarize this chart (ideally do a clearer graphic) and cite.

https://www.srgresearch.com/articles/aws-remains-dominant-despite-microsoft-and-google-growth-surges

More general overview of companies in AWS ecosystem.

Preferably a market map type graphic.

• Aggressive healthchecks
• Autoscaler termination policies tied to ELB healthcheck status

Elaborate, put in Gotcha section for ELBs

(Mentioned in review on #31)

Need to start this! (Under construction is fine.)

Links (what others?):
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
https://medium.com/cloud-uprising/using-lambda-backed-custom-resources-in-all-aws-regions-or-at-least-faking-it-part-1-intro-d679a50f7330#.szbzij9ok

Link to some good examples of using boto3 for automation, for the #boto section.

Would be nice to summarize SLAs, and may as well compare with GCP too (as sometimes this guides choices).

• Basic cost comparisons
• Put in context of modeling overall all-in costs including people and oppty costs

AWS' own Elasticsearch service is growing in popularity. Discuss basics, when to manage yourself vs with AWS' managed service, etc.

Blogs:

• A Comprehensive Guide to Building a Scalable Web App on Amazon Web Services, part 1
• 5 AWS Mistakes You Should Avoid (https://cloudonaut.io/5-aws-mistakes-you-should-avoid/)
• AWS Tips I Wish I'd Known Before I Started (https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/) and HN discussion (https://news.ycombinator.com/item?id=7172060)
• AWS EBS, Glacier And S3 Overview

Books:

• https://www.manning.com/books/amazon-web-services-in-action

How credits work and how to save money/use them. Credit expiration.

Merge all cost management into one place and forward link where appropriate.

@nitingithub to expand coverage.

https://aws.amazon.com/certificate-manager/

improve/update Lambda section and add other key serverless items there (or possibly in API Gateway or other sections).

Link to https://github.com/anaibol/awesome-serverless/ for more details but cover more important items.

Stuff to include:

• https://github.com/serverless/serverless
• https://github.com/Miserlou/Zappa
• https://github.com/awslabs/chalice

Links of interest:
http://www.slideshare.net/ServerlessConf/ben-kehoe-serverless-architecture-for-the-internet-of-things
https://github.com/awslabs/aws-iot-examples

Sketch out a diagram covering network costs:
Internet <-> EC2
Internet <-> CloudFront
EC2 <-> S3
region <-> region
AZ <-> AZ
VPC <-> VPC

EC2 details:
https://aws.amazon.com/ec2/pricing/#Data_Transfer

These are regional (not AZ specific) services with the same data transfer prices outbound (and zero inbound):
https://aws.amazon.com/s3/pricing/
https://aws.amazon.com/glacier/pricing/
https://aws.amazon.com/dynamodb/pricing/
https://aws.amazon.com/sqs/pricing/
https://aws.amazon.com/ses/pricing/
https://aws.amazon.com/simpledb/pricing/

These are AZ-specific services:
https://aws.amazon.com/rds/pricing/
https://aws.amazon.com/redshift/pricing/
https://aws.amazon.com/elasticache/pricing/

Other ways to get data in/out:
https://aws.amazon.com/cloudfront/pricing/
https://aws.amazon.com/directconnect/pricing/
https://aws.amazon.com/vpc/pricing/

Add basic info on these in the security section: AlertLogic, Coalfire, Dome9, etc.

https://aws.amazon.com/marketplace

• When to use
• How appliances work
• Major players
• Stats or overview

One blog with basics:
http://cloudacademy.com/blog/aws-security-iam-federated-access-trusted-advisor-and-consolidated-billing/

Most of these entries currently missing.

List of third-party tools is too short.

CloudCheckr, CloudHealth, Cloudability, Cloudaware, OpsClarity, etc.

Many listed here:
http://www.cloudaware.com/cloudaware-competitive-landscape/index.html

Worth linking to and covering a few high-level points:
https://d0.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf

Link to a couple of the best articles / getting started guides
Also key resources like awesome-aws, maybe also other ideas like meetup groups

Follow-on from #41. This is often noticed or discussed, but not sure it's widely understood.
Things to cover (with links):

• What is parallelized (prior to 2013 there wasn't parallelization)
• What CloudFormation scenarios can be slower than non-CloudFormation API usage?

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html

Tips -> EC2 Tips etc so URL anchors are unique and stable as we edit the guide.

Topics:

• Listing and pagination
• Need for multi-threaded S3 crawl over keys for speed
  • Prefix-based listings, with separators
  • Hash-type prefixes with known alphabet, uniform distribution
• Possibly: Reassigning work; using markers to optimize if alphabet is not known
• Tallying usage by mapreduce over keys that propagate usage up by folder

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

Cover some common approaches to backup filesystems to S3. See also #49.

Would be good to mention backup options/tools.

• https://github.com/zbackup/zbackup (deduplicating backups, inspired by rsync, in C++, analyzed here)
• https://github.com/restic/restic (deduplicating backups, in Go)
• https://github.com/borgbackup/borg (deduplicating backups, in Python, a fork of Attic with more active development)
• https://github.com/ncw/rclone (data sync to cloud)
• https://github.com/camlistore/camlistore

https://aws.amazon.com/s3/faqs/
https://aws.amazon.com/ec2/faqs/
etc.

• What certifications are available
• Who gets them

https://aws.amazon.com/certification/

At least mention major ones and who uses them, etc.
https://github.com/Netflix/asgard
http://www.spinnaker.io/

Include items from here:
https://cloud.google.com/docs/compare/aws/
https://storage.googleapis.com/gregsramblings-downloads/aws-gcp-mapping-full.pdf

For key sections where help/new ownership is needed, link to specific issues, not just contributing page.

For all topics where there are links to EC2, S3, etc, link to the section headings instead.

Start a section with "common patterns". These are general architectural patterns with links to useful blogs.

• Simple website
• Static website
• Log processing
• Analytics pipeline: Extend data source (e.g. Kinesis) and summarize data in Redshift
• High availability (within a region)
• Multi-region replication
• “serverless” deployment
  • lambda-chat example
• Security and IAM policies
• Source-driven infrastructure configuration
• A reasonable set of common-purpose buckets, like temp, user, amis, cloudfront content etc.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html