asascience / devops-tasks Goto Github PK

This is to standardize patching policy for AWS EC2 instances since currently there is no standard or schedule being followed by GTT for patching.

There were few issues with the pods failing on EKS cluster
Set up the cluster autoscaler to adjust the nodes in the cluster when pods fail or are rescheduled onto other nodes

@jonmjoyce

I have another task when you get time, not urgent but just trying to reduce our EC2 costs. Could you change out the node group for the nextgen-dev cluster to m6i.large instance type, running 2 of them? Currently we are using 2 m5.xlarge but we don't need that many resources right now. Usually I create a new node group. Then to get the tags properly applied to the instances you edit the autoscaling group that gets created to add all of our required custom tags. Then if that all goes well you should be able to remove the old node group and the pods/resources should transfer over. There might be another way to do it but that's been my method. Thanks!

Working on setting up PRTG monitoring

• data pulled in the last 30 hours
• http monitor: query to make sure the service is up and running
• CPU utilization,
• Memory usage
• Low disk space alert
• URL monitoring
• Uptime/reboot events

Need to figure out where these TLS 1.0/1.1 requests are coming from so we can make updates where needed.

S3 bucket: https://s3.console.aws.amazon.com/s3/buckets/oceansmap

We have identified TLS 1.0 or TLS 1.1 connections to Amazon Simple Storage Service (Amazon S3) objects hosted in your account, which must be immediately updated for these connections to maintain their access to your S3 objects. Please update your client software as soon as possible to use TLS 1.2 or higher to avoid an availability impact. We recommend considering the time needed to verify your changes in a staging environment before introducing them into production.

As of June 28, 2023, we have begun deploying updates to the TLS configuration for all AWS API endpoints to a minimum of version TLS 1.2 even if you still have connections using these versions. These deployments will complete by no later than December 31, 2023. This update removes the ability to use TLS versions 1.0 and 1.1 with all AWS APIs in all AWS Regions [1].

What actions can I take to maintain access?
To avoid potential interruption, you must update all client software accessing your Amazon S3 objects using TLS 1.0 or 1.1, to use TLS 1.2 or higher. If you are unable or would prefer to not update all impacted clients, we recommend replacing direct client access to the S3 objects with use of a proxy, such as an Amazon CloudFront distribution. This will allow clients to access your S3 objects via Amazon CloudFront using any TLS version you choose to allow. Amazon CloudFront will forward the calls to your S3 objects using TLS 1.2 or higher. For more guidance for how to setup your CloudFront distribution to front your S3 object access, please review this Knowledge Center article [2].

How can I determine the client(s) I need to update?
We have provided the affected S3 bucket(s) in your account following this messaging. In order to gather additional information about the affected objects and user agents performing these calls, we recommend enabling Amazon CloudTrail data events on the affected S3 bucket(s) [3] [4]. The information contained in the S3 data events will help you pinpoint your client software that is responsible for using TLS 1.0 or TLS 1.1, so you may update it accordingly. Additionally, our related AWS Security blog post [1] provides information on how you may use TLS information in the CloudTrail tlsDetails field. Please note there is an associated cost for enabling CloudTrail data events, please see the CloudTrail pricing page for more detail [5]. Another alternative is to use Amazon S3 server-access logs, see the S3 Logging options page for more details and pricing information [6].

How can I enforce connections to my bucket(s) be over TLSv1.2 and above?
As a best practice, and to prepare for our enforcement of TLS 1.2 or higher, we recommend you proactively enforce a minimum of TLS 1.2 directly on all of your shared S3 bucket(s). You may do this by applying a bucket policy with the s3:TlsVersion condition key as per the documented this Knowledge Center article [7]

Connections details will be in the following format:
Region | Bucket name(s) | APIAction | TLSVersion | NumCalls | UserAgent
us-east-1 | oceansmap | REST.GET.BUCKET | TLSv1 | 1 | []

We have several machines in AWS with either missing or incorrect tags. Specifically the tags OS and Classification which control how the firewall rules apply to that machine. For example, the machine MapApp OM3 demo's classification is currently "Test" but it should be "DMZ Test". The OS tag allows the machine access to update repositories, and the Classification allows traffic into the machine.

If it's obvious what the tag should be could you please go through and update them? If unsure contact the POC if you know who it is.

Longer term: I remember we talked briefly when you first started about a process for enforcing tags but maybe we aren't ready to take it that far. Maybe for now we could create a script that scans the machines for invalid tags so we're aware of them.

-There were issues with python dependencies (~20) due to obsolete/incompatible versions

• Had to update recent or compatible versions and update requirements file
• Build and deploy are now working as expected

Currently GitHub is on a legacy plan and pre-requisite is to trim down the user list on GitHub before it can be upgraded to the right plan. Trying to archive or delete unused repos by coordinating with the team.
@benjwadams @patrick-tripp @mpiannucci @jonmjoyce @DonaldMoretti

-Configured application load balancer to overcome issues with ip changes or re-deploy

We already have a service running on nextgen-dev Kubernetes which is hosting the XREDS website: https://nextgen-dev.ioos.us/xreds/

Currently this service is manually deployed by @mpiannucci.

Source code and instructions for deploying xreds: https://github.com/asascience-open/xreds

We would like to create a CI pipeline to build this project. We can also explore CD options but for now a simple build process and one-line to deploy is fine without taking too much time. Please work with @mpiannucci if you have questions configuring the build process.