Code Monkey home page Code Monkey logo

cdk-athena-quicksight's Introduction

QuickSight Fetcher CLI

Overview

This Python CLI script provides functionality to interact with Amazon QuickSight. It allows you to fetch QuickSight resource descriptions and clone dashboards from a specified AWS account.

Prerequisites

  • Python 3.x
  • AWS CLI & Profile configured with necessary permissions

Installation

Clone the repository:

git clone https://github.com/jrsalgado/cdk-athena-quicksight

Create a virtual environment:

python3 -m venv .venv
source .venv/bin/activate

Install dependencies:

pip install --editable .

Disaster Recovery

By fetching and storing resource descriptions, users can create backups of QuickSight configurations, dashboards, analyses, data sets, data sources, Athena workgroups, Athena data catalogs, and Glue databases.

  1. Fetch Resources: Use qscli fetch all to store descriptions of all resources in the /infra_base directory.

  2. Generate Templates: Use qscli build commands to create CloudFormation templates for each resource type.

  3. Deploy Templates: Deploy CloudFormation templates with qscli deploy to recreate QuickSight environment and resources. Customize configurations using parameter override files.

This process ensures swift recovery of QuickSight, Athena and Glue resources in case of a disaster.

CLI Commands

Fetch Resource Descriptions

Command Description Status
fetch all Fetch all related resources from AWS Account ✅
fetch dashboard Fetch a Dashboard from its ID TODO
fetch dashboards Fetch all Dashboards ✅
fetch analysis Fetch an Analysis from its ID TODO
fetch analyses Fetch all Analyses ✅
fetch data-set Fetch a Data Set from its ID TODO
fetch data-sets Fetch all Data Sets ✅
fetch data-source Fetch a Data Source from its ID TODO
fetch data-sources Fetch all Data Sources ✅
fetch workgroup Fetch an Athena Workgroups by name TODO
fetch workgroups Fetch all Athena Workgroups ✅
fetch data-catalog Fetch an Athena Data Catalogs by name TODO
fetch data-catalogs Fetch all Athena Data Catalogs ✅
fetch glue Fetch all Glue Databases ✅
fetch lambdas Fetch all Lambdas descriptions ✅

Build CloudFormation Template

Command Description Status
build dashboard Build a template for a specific Dashboard by ID, and its dependencies. ✅
build dashboards Build templates for all Dashboards. TODO
build analysis Build a template for a specific Analysis by ID. ✅
build analyses Build templates for all Analyses. TODO
build data-set Build a template for a specific Data Set by ID. ✅
build data-sets Build templates for all Data Sets. TODO
build data-source Build a template for a specific Data Source by ID. ✅
build data-sources Build templates for all Data Sources. TODO
build athena Build a template for a specific Athena Workgroup and Data Catalogs. ✅
build glue Build a template for a specific Glue Database ✅

Deploy CloudFormation Template

Command Description Status
deploy dashboard Deploy a single Dashboard template ✅
deploy dashboards Deploy all dashboards templates TODO
deploy analysis Deploy a single Analysis template ✅
deploy analyses Deploy all Analyses TODO
deploy dataset Deploy a single Data Set template ✅
deploy datasets Deploy all Data Sets TODO
deploy datasource Deploy a single Data Source template ✅
deploy datasources Deploy all Data Sources TODO
deploy athena Deploy Athena resources ✅
deploy glue Deploy Glue Database ✅

Usage

Fetch Resources

This command fetches and stores the descriptions of all the necessary resources of a specific AWS account on the /infra_base directory.

qscli fetch all --account-id <ACCOUNT_ID> --profile <PROFILE_NAME>

If already authenticated (Cloud9), the option --profile can be skipped

Athena

Build Athena Template

This command generates a CloudFormation template to clone an existing Athena Workgroup and Data Catalog.

Get the name of the desired athena resource from:

  • workgroup: /infra_base/XXXXXXXXXXX/athena/workgroups/
  • data-catalog: /infra_base/XXXXXXXXXXX/athena/data-catalog/
qscli build athena --workgroup-name <WORKGROUP_NAME> --catalog-name <DATA_CATALOG_NAME> --account-id <ACCOUNT_ID>

Deploy Athena Template

qscli deploy athena <RELATIVE_PATH_TO_TEMPLATE> --account-id <ACCOUNT_ID> --profile <AWS_PROFILE> --parameters-path <PARAMS_FILE_NAME>

The path to the template to deploy should be:

  • ./CFTemplates/XXXXXXXXXXX/athena/athena_template.yaml

--parameters-path (default: athena.local.txt)

  • Create a new file in /parameter-overrides with the following values
AthenaWorkgroupName=<REPLACE>             # Name for the new Workgroup
AthenaWorkgroupOutputLocation=<REPLACE>   # S3 bucket location to store query results
AthenaDataCatalogName=<REPLACE>           # Name for the new Data Catalog

Glue

Build Glue Template

This command generates a CloudFormation template to clone an existing Glue Database and its tables.

Get the name of the desired glue database name from:

  • /infra_base/XXXXXXXXXXX/glue/databases/
qscli build glue <DATABASE_NAME> --account-id <ACCOUNT_ID>

Deploy Glue Template

qscli deploy glue <RELATIVE_PATH_TO_TEMPLATE> --account-id <ACCOUNT_ID> --profile <AWS_PROFILE> --parameters-path <PARAMS_FILE_NAME>

The path to the template to deploy should be:

  • ./CFTemplates/XXXXXXXXXXX/glue/<DATABASE_NAME>.yaml

--parameters-path (default: glue.local.txt)

  • Create a new file in /parameter-overrides with the following values
GlueDatabaseName=<REPLACE>

Quicksight

For Quicksight resources to be cloned properly, Glue and Athena resources must be created first.

Build Dashboard Template

This command generates a CloudFormation template to clone an existing dashboard from its ID.

Get the ID of the desired dashboard from /infra_base/XXXXXXXXXXX/dashboards/list-dashboards.yaml

qscli build dashboard <DASHBOARD_ID> --account-id <ACCOUNT_ID> --create-dependencies True

--create-dependencies True Finds and create all the quicksight resources that are needed to create a fully working Dashboard

Deploy Dashboard Template

qscli deploy dashboard <RELATIVE_PATH_TO_TEMPLATE> --account-id <ACCOUNT_ID> --region <REGION> --profile <PROFILE_NAME> --parameters-path <PARAMS_FILE_NAME>

The path to the template to deploy should be:

  • ./CFTemplates/XXXXXXXXXXX/dashboards/<DASHBOARD_ID>.yaml

Command options

  • --account-id [Required]

  • --region (default: us-east-1)

  • --profile (Optional if already authenticated)

  • --parameters-path (default: dashboard.local.txt)

    • Create a new file in /parameter-overrides with the following values
DataSourceId01=<REPLACE>            # Has to be unique
DataSourceName01=<REPLACE>
DataSourceAthenaWorkGroup01=<REPLACE> # Name of the new workgroup from Athena

DataSetAthenaId01=<REPLACE>         # Has to be unique
DataSetAthenaName01=<REPLACE>
DataSetAthenaCatalog01=<REPLACE>    # Name of the new Athena Data Catalog
DataSetAthenaTableName01=<REPLACE>  # Name of the table inside Glue Database
DataSetAthenaSchema01=<REPLACE>     # Name of the Glue Database

DashboardId01=<REPLACE>             # Has to be unique
DashboardName01=<REPLACE>
DashboardDataSetIdentifier01=<REPLACE with Data Set name> # Has to be the dataset name

AnalysisId01=<REPLACE>              # Has to be unique
AnalysisName01=<REPLACE>

QuickSightUsername=<REPLACE from original> # Quicksight username with access

For building and deploying other Quicksight resources, the process is the same

  • Fetch all resources.
  • Generate a CF template using the build command and providing the id for the resource to clone, (with --create-dependencies True to automatically resolve all of the dependencies).
  • Create the parameters override file with the desired values for the specific resource.
  • And use the deploy command to create the stack on the desired AWS account.

cdk-athena-quicksight's People

Contributors

jrsalgado avatar

Watchers

avatar avatar

cdk-athena-quicksight's Issues

Requerimientos Iniciales

Requerimientos Principales:

  1. Acceso al Repositorio de Git:

    • Acceso al repositorio de Git con los permisos adecuados.

  2. Suscripción a Quicksight:

    • Se requiere una suscripción a Quicksight para facilitar la visualización de datos y poder desplegar el template de cloudformation

  3. Esquemas de Tablas de Athena:

    • Se necesita saber el nombre de los esquemas de athena
    • Son las que aparecian en el diagrama?
      • Compliance
      • RSS
      • CIS
      • Managed

  4. Acceso a Ambiente de Sandbox:

    • Garantizar el acceso al ambiente de sandbox para pruebas y desarrollo.

Requerimientos de Código:

  1. Variables de Entorno:

    • Identificar y documentar las variables de entorno necesarias para la configuración del proyecto.

  2. Workgroup de Athena:

    • Proporcionar detalles del workgroup de Athena, incluyendo ARN, ID y nombre principalmente.

  3. Región, Account ID:

    • Especificar la región y el Account ID asociados al proyecto.

  4. Nombres de Base de Datos y Tablas en Athena:

    • Confirmar que los nombres de la base de datos y las tablas para los datasets en Athena coincidan con las anteriormente mencionadas (compliance, rss, cis, managed).

Tareas para el Equipo:

  1. Importar Workgroup de Athena a Template de CloudFormation:

    • Integrar el workgroup de Athena en el template de CloudFormation.

  2. Crear DataSource para el Workgroup de Athena:

    • Desarrollar el DataSource necesario para la integración del workgroup de Athena.

  3. Crear Datasets para Esquemas Específicos:

    • Generar los datasets correspondientes a los esquemas mencionados (compliance, rss, cis, managed).

  4. Ajustar Secciones de CloudFormation:

    • Realizar ajustes necesarios en las secciones pertinentes del template de CloudFormation(Mappings, Outputs, Conditions).

  5. Pruebas en Ambiente de Sandbox:

    • Realizar pruebas exhaustivas en el ambiente de sandbox para garantizar la estabilidad y funcionalidad del sistema.

  6. Crear Más Tareas:

    • Identificar y documentar tareas adicionales necesarias para el progreso continuo del proyecto.

CF Template - Create Necessary Params

Investigate common params and create their CF Params section according to common santander Env params and specific project params, this params are required to deploy on Sanbox(Fidel account) environment and/or Santander, some of this params may be resolved by Edgar

Parameters:
  Project:
    Type: String
    Default: test
    Description: Name of the project for the tag
  Environment:
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - qa
      - stg
      - pre
      - ocu
      - sha
      - prod
    Description: Name of the environment for the tag
  Country:
    Type: String
    AllowedValues:
      - es
      - de
      - pt
      - nl
      - ar
      - mc
      - mx
      - us
    Description: Name of the country for the tag
  QuickSightUsername:
    Type: String
    Default: master
    ConstraintDescription: QuickSight username must be between 5 and 20 characters
    Description: Quicksight username
    MaxLength: 20
    MinLength: 5
  AthenaDatabaseName:
    Type: String
    Default: santander
    Description: Athena Database Name used for QS dataset queries
  AthenaWorkgroupName:
    Type: String
    Description: Athena Workgorup name

Data - Gather Datasource AWS values

Investigate data source AWS data from atthena to be used to create Quicksight datasources for Sandbox and Santander environments
Edgar(ask santander)
Fidel (resolve from his account)
This data is required to create resources and create parameters to deploy on each environment

Vars: - AWS::AccountId
- AthenaWorkgroupName
- AWS::Region
- QuickSightUsername

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.