nehasingh110 Goto Github PK

cedar

Charts for ArcGIS GeoServices

taxi-analysis

Big Data Analysis on NYC Taxi in MapReduce/Hadoop

udacity-project-1-data-modelling-with-postgres

udacity-project-2-data-warehouse

Loaded data from S3 to staging tables on Redshift and executed SQL statements that create the analytics tables from these staging tables.

udacity-project-3-data-lake

Loaded data from S3, processed the data into analytics tables using Spark, and loaded them back into S3. Deployed this Spark process on a cluster using AWS.

udacity-project-4-airflow-data-pipelines

Developed high grade data pipelines that are dynamic and built from reusable tasks, can be monitored, and allow easy backfills. Developed tests against the datasets after the ETL steps have been executed to catch any discrepancies in the datasets. Developed custom operators to perform tasks such as staging the data, filling the data warehouse, and running checks on the data. The source datasets consisted of JSON logs that tell about user activity in the application and JSON metadata about the songs the users listen to. This data was moved from S3 to a data warehouse in Amazon Redshift for further processing.

udacity-project-5-capstone

This project includes reading i94 immigration data in parquet file format and US demographics data in csv format, exploring these 2 data sets, making ammendments, making the 2 sets compatible so that they can be joined later for generating reports and finally writing out this data to S3 and then to Redshift tables.